Skip to content

Commit

Permalink
set kubelet's --provider-id flag
Browse files Browse the repository at this point in the history
- What I did
I Added AWS specific systemd unit (aws-kubelet-providerid.service) and file (/usr/local/bin/aws-kubelet-providerid) for generating the AWS instance provider-id (then stored in the KUBELET_PROVIDERID env var), in order to pass it as the --provider-id argument to the kubelet service binary.
We needed to add such flag, and make it non-empty only on AWS, to make the node syncing (specifically backing instance detection) work via provider-id detection, to cover cases where the node hostname doesn't match the expected private-dns-name (e.g. when a custom DHCP Option Set with empty domain-name is used).

Should fix: https://bugzilla.redhat.com/show_bug.cgi?id=2084450
Reference to an upstream issue with context: kubernetes/cloud-provider-aws#384

- How to verify it
Try the reproduction steps available at: https://bugzilla.redhat.com/show_bug.cgi?id=2084450#c0 while launching a cluster with this MCO PR included.
Verify that the issue is not reproducible anymore.
  • Loading branch information
damdo committed May 25, 2022
1 parent b52e75e commit ae86af7
Show file tree
Hide file tree
Showing 4 changed files with 51 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
mode: 0755
path: "/usr/local/bin/aws-kubelet-providerid"
contents:
inline: |
#!/bin/bash
set -e -o pipefail
NODECONF=/etc/systemd/system/kubelet.service.d/20-aws-providerid.conf
if [ -e "${NODECONF}" ]; then
echo "Not replacing existing ${NODECONF}"
exit 0
fi
# Due to a potential mismatch between Hostname and PrivateDNSName with clusters that use custom DHCP Option Sets
# which can cause issues in cloud controller manager node syncing
# (see: https://github.com/kubernetes/cloud-provider-aws/issues/384),
# set KUBELET_PROVIDERID to be a fully qualified AWS instace provider id.
# This new variable is later used to populate the kubelet's `provider-id` flag, later set on the Node .spec
# and used by the cloud controller manager's node controller to retrieve the Node's backing instance.
# This is obtained by using afterburn service variables, in turn obtained from metadata retrival.
# See respective systemd unit metadata related afterburn doc: https://coreos.github.io/afterburn/usage/attributes/
cat > "${NODECONF}" <<EOF
[Service]
Environment="KUBELET_PROVIDERID=aws://${AFTERBURN_AWS_AVAILABILITY_ZONE}/${AFTERBURN_AWS_INSTANCE_ID}"
EOF
23 changes: 23 additions & 0 deletions templates/common/aws/units/aws-kubelet-providerid.service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: aws-kubelet-providerid.service
enabled: true
contents: |
[Unit]
Description=Fetch kubelet provider id from AWS Metadata
# Run afterburn service for collect info from metadata server
# see: https://coreos.github.io/afterburn/usage/attributes/
Requires=afterburn.service
After=afterburn.service
# Wait for NetworkManager to report it's online
After=NetworkManager-wait-online.service
# Run before kubelet
Before=kubelet.service
[Service]
EnvironmentFile=/run/metadata/afterburn
ExecStart=/usr/local/bin/aws-kubelet-providerid
Type=oneshot
[Install]
WantedBy=network-online.target
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ contents: |
--volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
{{cloudConfigFlag . }} \
--hostname-override=${KUBELET_NODE_NAME} \
--provider-id=${KUBELET_PROVIDERID} \
--register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
--pod-infra-container-image={{.Images.infraImageKey}} \
--system-reserved=cpu=${SYSTEM_RESERVED_CPU},memory=${SYSTEM_RESERVED_MEMORY} \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ contents: |
--cloud-provider={{cloudProvider .}} \
{{cloudConfigFlag . }} \
--hostname-override=${KUBELET_NODE_NAME} \
--provider-id=${KUBELET_PROVIDERID} \
--pod-infra-container-image={{.Images.infraImageKey}} \
--system-reserved=cpu=${SYSTEM_RESERVED_CPU},memory=${SYSTEM_RESERVED_MEMORY} \
--v=${KUBELET_LOG_LEVEL}
Expand Down

0 comments on commit ae86af7

Please sign in to comment.