Harvester cloud provider stuck in containercreating due to error with cloug-config hostpath mount.

[harrisonbc]

I am trying to create a downstream RKE2 cluster on Harvester using cluster-templates.

I created a values.yaml file and installed using the helm command.

The Node pool is created, the node VM, the IP address is retrieved and RKE2 is installed.

I can access the new cluster as it is initialising:

kubectl get pods -A
NAMESPACE       NAME                                                    READY   STATUS              RESTARTS   AGE
cattle-system   cattle-cluster-agent-5d8b874678-xrkcj                   0/1     Pending             0          7m29s
kube-system     etcd-rke2-single-all-j9lbx-9n6kg                        1/1     Running             0          7m8s
kube-system     harvester-cloud-provider-7bc65f5fcf-9zkmh               0/1     ContainerCreating   0          7m4s
kube-system     harvester-csi-driver-controllers-675bcdb6b8-2dzcf       0/3     Pending             0          7m12s
kube-system     harvester-csi-driver-controllers-675bcdb6b8-2ksq4       0/3     Pending             0          7m12s
kube-system     harvester-csi-driver-controllers-675bcdb6b8-m8kxn       0/3     Pending             0          7m12s
kube-system     helm-install-harvester-cloud-provider-w86tl             0/1     Completed           0          7m31s
kube-system     helm-install-harvester-csi-driver-62hgl                 0/1     Completed           0          7m31s
kube-system     helm-install-rke2-canal-8pw92                           0/1     Completed           0          7m31s
kube-system     helm-install-rke2-coredns-nl24l                         0/1     Completed           0          7m31s
kube-system     helm-install-rke2-ingress-nginx-l8ktb                   0/1     Pending             0          7m31s
kube-system     helm-install-rke2-metrics-server-clmbq                  0/1     Pending             0          7m30s
kube-system     helm-install-rke2-snapshot-controller-crd-q8rvq         0/1     Pending             0          7m30s
kube-system     helm-install-rke2-snapshot-controller-wq87s             0/1     Pending             0          7m30s
kube-system     helm-install-rke2-snapshot-validation-webhook-ngjqv     0/1     Pending             0          7m30s
kube-system     kube-apiserver-rke2-single-all-j9lbx-9n6kg              1/1     Running             0          7m27s
kube-system     kube-controller-manager-rke2-single-all-j9lbx-9n6kg     1/1     Running             0          7m4s
kube-system     kube-proxy-rke2-single-all-j9lbx-9n6kg                  1/1     Running             0          7m25s
kube-system     kube-scheduler-rke2-single-all-j9lbx-9n6kg              1/1     Running             0          7m
kube-system     rke2-canal-vtht8                                        2/2     Running             0          6m59s
kube-system     rke2-coredns-rke2-coredns-5b79675658-c6b9b              0/1     Pending             0          7m2s
kube-system     rke2-coredns-rke2-coredns-autoscaler-79c9dc4c88-l2ssd   0/1     Pending             0          7m2s

As you can see the harvester-cloud-provider is stuck in container creating.

I then described the pod and got:

kubectl describe  pods harvester-cloud-provider-7bc65f5fcf-9zkmh   -n kube-system
Name:             harvester-cloud-provider-7bc65f5fcf-9zkmh
Namespace:        kube-system
Priority:         0
Service Account:  harvester-cloud-provider
Node:             rke2-single-all-j9lbx-9n6kg/192.168.1.208
Start Time:       Tue, 04 Feb 2025 14:57:03 +0000
Labels:           app.kubernetes.io/instance=harvester-cloud-provider
                  app.kubernetes.io/name=harvester-cloud-provider
                  pod-template-hash=7bc65f5fcf
Annotations:      <none>
Status:           Pending
IP:               192.168.1.208
IPs:
  IP:           192.168.1.208
Controlled By:  ReplicaSet/harvester-cloud-provider-7bc65f5fcf
Containers:
  harvester-cloud-provider:
    Container ID:  
    Image:         docker.io/rancher/harvester-cloud-provider:v0.2.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      harvester-cloud-provider
    Args:
      --cloud-config=/etc/kubernetes/cloud-config
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/kubernetes/cloud-config from cloud-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dtphm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  cloud-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/cloud-config
    HostPathType:  File
  kube-api-access-dtphm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 cattle.io/os=linux:NoSchedule
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/etcd:NoExecute
                             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  7m14s                 default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         6m27s                 default-scheduler  Successfully assigned kube-system/harvester-cloud-provider-7bc65f5fcf-9zkmh to rke2-single-all-j9lbx-9n6kg
  Warning  FailedMount       15s (x11 over 6m27s)  kubelet            MountVolume.SetUp failed for volume "cloud-config" : hostPath type check failed: /etc/kubernetes/cloud-config is not a file

I looked at the host path and found that /etc/kubernetes/cloud-config does not exist. (or indeed the /etc/kubernetes directory)
I have added the values.yaml file (as a .txt)

single-node.yaml.txt

Am I doing something wrong?

[harrisonbc]

If I copy the harvester kubeconfig file to /etc/kubernetes/cloud-config the process completes and the cluster is created successfully.

Is there a step missing in the setup for this to be done automatically?