Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes node add failed! #36

Open
CYS158 opened this issue Jun 16, 2022 · 5 comments
Open

Kubernetes node add failed! #36

CYS158 opened this issue Jun 16, 2022 · 5 comments
Assignees
@cloudnativer
Labels
question Further information is requested

Comments

@CYS158
Copy link

CYS158 commented Jun 16, 2022

add node failed
logs：

[Info] 2022-06-16 16:58:14.38258995 +0800 CST m=+1655.018892666 Adding kubernetes node, please wait ...

Kubernetes Cluster Label: K8s-Cluster1
Kubernetes Node: 8.8.88.86
SSH Operation Port: 22
Operating System Type: rhel7
Automatically Upgrade OS Kernel: NO
System User for Operation: root

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : 0.Distributing deployment files to target host, please wait...] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : copy] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/genfile : copy] ***
changed: [8.8.88.86]

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 1.Create base package directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 2.Perform installation pretreatment] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : shell] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 3.Turn off firewall] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : service] ***
fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service ufw: host"}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : service] ***
fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service SuSEfirewall2: host"}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 4.Turn off selinux] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 5.Turn off postfix] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 6.Set system limit] ***
changed: [8.8.88.86] => (item=* soft nofile 65536)
changed: [8.8.88.86] => (item=* hard nofile 65536)
changed: [8.8.88.86] => (item=* soft nproc 65536)
changed: [8.8.88.86] => (item=* hard nproc 65536)
changed: [8.8.88.86] => (item=* soft memlock unlimited)
changed: [8.8.88.86] => (item=* hard memlock unlimited)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 7.Set ipvs_modules] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 8.Run ipvs_modules] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 9.Add sysctl parameter configuration] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 10.Set sysctl parameter configuration] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 11.Create journal directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 12.Create journal.conf.d directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 13.Add prophet file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 14.Start systemd-journald service] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 15.Install TTY software] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : copy] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 16.Create resolv config file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 17.Create k8s cert directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 18.Decompress k8s software package] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 19.Decompress cfssl software package] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/all : 20.Decompress helm software package] ***
changed: [8.8.88.86]

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 1.Cleaning up garbage files left in history.] ***
fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : service] ***
fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service kube-proxy: host"}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : service] ***
fatal: [8.8.88.86]: FAILED! => {"changed": false, "msg": "Could not find the requested service containerd: host"}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : shell] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86] => (item=containerd)
ok: [8.8.88.86] => (item=containerd-shim)
ok: [8.8.88.86] => (item=containerd-shim-runc-v1)
ok: [8.8.88.86] => (item=containerd-shim-runc-v2)
ok: [8.8.88.86] => (item=nerdctl)
ok: [8.8.88.86] => (item=ctr)
ok: [8.8.88.86] => (item=runc)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 2.Create CNI plug-in directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 3.Decompress CNI plug-in package] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : unarchive] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 4.Create runtime config dir] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 5.Decompress runtime software package] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : template] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 6.Create containerd service] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 7.Create ipvsinit service] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 8.Create ipvsinit shell] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 9.Enable ipvsinit service] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 10.Start runtime service] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 11.Decompress runtime images package] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 12.Load runtime images] ***
changed: [8.8.88.86] => (item=metrics-server-amd64.tar)
changed: [8.8.88.86] => (item=metrics-scraper.tar)
changed: [8.8.88.86] => (item=heapster-amd64.tar)
changed: [8.8.88.86] => (item=pause-amd64.tar)
changed: [8.8.88.86] => (item=traefik.tar)
changed: [8.8.88.86] => (item=alpine.tar)
changed: [8.8.88.86] => (item=kubernetes-dashboard-v2.4.0-amd64.tar)
changed: [8.8.88.86] => (item=coredns.tar)
changed: [8.8.88.86] => (item=registry.tar)
changed: [8.8.88.86] => (item=calico-cni.tar)
changed: [8.8.88.86] => (item=calico-kube-controllers.tar)
changed: [8.8.88.86] => (item=calico-node.tar)
changed: [8.8.88.86] => (item=calico-pod2daemon-flexvol.tar)
changed: [8.8.88.86] => (item=flannel-cni-plugin.tar)
changed: [8.8.88.86] => (item=flannel.tar)
changed: [8.8.88.86] => (item=kube-router.tar)
changed: [8.8.88.86] => (item=weave-kube.tar)
changed: [8.8.88.86] => (item=weave-npc.tar)
changed: [8.8.88.86] => (item=cilium.tar)
changed: [8.8.88.86] => (item=cilium-operator-generic.tar)

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : 13.Set runtime images tag and create pushsysimages shell] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000base/runtime : shell] ***
changed: [8.8.88.86] => (item=metrics-server-amd64:v0.5.0)
changed: [8.8.88.86] => (item=metrics-scraper:v1.0.7)
changed: [8.8.88.86] => (item=heapster-amd64:v1.5.4)
changed: [8.8.88.86] => (item=pause-amd64:3.5)
changed: [8.8.88.86] => (item=traefik:v2.0.7)
changed: [8.8.88.86] => (item=alpine:3.6)
changed: [8.8.88.86] => (item=kubernetes-dashboard-amd64:v2.4.0)
changed: [8.8.88.86] => (item=coredns:1.3.1)
changed: [8.8.88.86] => (item=registry:v2.7.1)
changed: [8.8.88.86] => (item=calico-node:v3.19.3)
changed: [8.8.88.86] => (item=calico-pod2daemon-flexvol:v3.19.3)
changed: [8.8.88.86] => (item=calico-cni:v3.19.3)
changed: [8.8.88.86] => (item=calico-kube-controllers:v3.19.3)
changed: [8.8.88.86] => (item=flannel-cni-plugin:v1.0.0)
changed: [8.8.88.86] => (item=flannel:v0.15.1)
changed: [8.8.88.86] => (item=kube-router:v1.3.2)
changed: [8.8.88.86] => (item=weave-kube:2.8.1)
changed: [8.8.88.86] => (item=weave-npc:2.8.1)
changed: [8.8.88.86] => (item=cilium:v1.9.0)
changed: [8.8.88.86] => (item=cilium-operator-generic:v1.9.0)

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 1.Distribution kubelet cert] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 2.Clean up invalid certificates] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 3.Copy kubelet config and shell file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : copy] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 4.Set IPVS rules] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 5.Create /home/kube-install/k8s/kubernetes directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 6.Create kubelet service startup file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kubelet : 7.Start kubelet service] ***
changed: [8.8.88.86]

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 1.Create kube-proxy.config.yaml configuration] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 2.Create /home/kube-install/k8s/kubernetes/kube-proxy directory] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 3.Create kube-proxy service startup file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/kube-proxy : 4.Start kube-proxy service] ***
changed: [8.8.88.86]

PLAY [master1] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 1.Enable csr-crb configuration] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 2.Wait 90s] ***
ok: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 3.Approve node csr] ***
fatal: [8.8.88.81]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/kubectl get csr | awk '/Pending/ {print $1}' | xargs kubectl certificate approve", "delta": "0:00:00.118209", "end": "2022-06-16 17:06:44.673905", "msg": "non-zero return code", "rc": 123, "start": "2022-06-16 17:06:44.555696", "stderr": "error: one or more CSRs must be specified as or -f ", "stderr_lines": ["error: one or more CSRs must be specified as or -f "], "stdout": "", "stdout_lines": []}
...ignoring

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 4.Wait 30s] ***
ok: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : 5. Setting scheduling label for k8s node] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x0000000000node/approve-csr : shell] ***
changed: [8.8.88.81]

PLAY [addnode] *****************************************************************

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : Create reboot config file] ***
changed: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : Congratulations, node successfully joined kubernetes cluster! "The operating system will automatically restart to take effect on the cluster configuration."] ***
ok: [8.8.88.86]

TASK [/root/kube-install/data/output/K8s-Cluster1/sys/0x00000000finish/addnode : shell] ***
changed: [8.8.88.86]

PLAY RECAP *********************************************************************
8.8.88.81 : ok=11 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
8.8.88.86 : ok=77 changed=62 unreachable=0 failed=0 skipped=0 rescued=0 ignored=5

[Error] 2022-06-16 17:13:19.840843904 +0800 CST m=+2560.477146626 Kubernetes node add failed!
@cloudnativer
Copy link
Owner

What is the detailed version of the operating system, the CNI plug-in selected, and the kube-install version?
Is there any restriction on the network from the k8s-master to the k8s-node?

@cloudnativer
Copy link
Owner

Has this k8s-node performed the action of adding repeatedly? You can try delnode and addnode again.

@CYS158
Copy link
Author

CYS158 commented Jun 19, 2022

操作系统的详细版本，选择的CNI插件，kube-install版本是多少？ 从 k8s-master 到 k8s-node 的网络有什么限制吗？

CentOS Linux release 7.9.2009 (Core)
kube-install-allinone-v0.8.0.tgz
new centos7
systemctl stop firewalld

@CYS158
Copy link
Author

CYS158 commented Jun 19, 2022

这个k8s-node是否执行了重复添加的动作？您可以再次尝试 delnode 和 addnode。

New system, first execution addnode

@houseonline
Copy link
Collaborator

houseonline commented Jun 21, 2022 •
edited
Loading

Execute kubectl get cs,node -o wide on the command line to see whether the cluster information and node information are running normally??

@houseonline houseonline added the question Further information is requested label Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cloudnativer cloudnativer

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@houseonline @CYS158 @cloudnativer