k8s集群node上磁盘满了,导致其上新建pod一直处于ContainerCreating状态,describe pod如下:
Warning FailedCreatePodSandBox 14m (x17446 over 10h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c3e5a072c9dc907b85cd6dc74b41a86705d18dc7d27450f79aee451f4d25d3c6" network for pod "kafka-client-provisioner-5699f4779d-bhdmr": networkPlugin cni failed to set up pod "kafka-client-provisioner-5699f4779d-bhdmr_kafka" network: failed to read MTU file: strconv.Atoi: parsing "": invalid syntax
Normal SandboxChanged 4m19s (x17818 over 10h) kubelet Pod sandbox changed, it will be killed and re-created
journalctl -xe日志如下:
5月 11 21:14:41 k8s kubelet[21929]: E0511 21:14:41.607022 21929 cni.go:361] "Error adding pod to network" err="failed to read MTU file: strconv.Atoi: parsing \"\": invalid syntax" pod="kafka/kafka-client-provisioner-5699f4779d-bhdmr" podSandboxID={Type:docker ID:ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b} podNetnsPath="/proc/302687/ns/net" networkType="calico" networkName="k8s-pod-network"
5月 11 21:14:41 k8s kubelet[21929]: E0511 21:14:41.987092 21929 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to set up sandbox container \"ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b\" network for pod \"kafka-client-provisioner-5699f4779d-bhdmr\": networkPlugin cni failed to set up pod \"kafka-client-provisioner-5699f4779d-bhdmr_kafka\" network: failed to read MTU file: strconv.Atoi: parsing \"\": invalid syntax"
5月 11 21:14:41 k8s kubelet[21929]: E0511 21:14:41.987134 21929 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to set up sandbox container \"ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b\" network for pod \"kafka-client-provisioner-5699f4779d-bhdmr\": networkPlugin cni failed to set up pod \"kafka-client-provisioner-5699f4779d-bhdmr_kafka\" network: failed to read MTU file: strconv.Atoi: parsing \"\": invalid syntax" pod="kafka/kafka-client-provisioner-5699f4779d-bhdmr"
5月 11 21:14:41 k8s kubelet[21929]: E0511 21:14:41.987165 21929 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to set up sandbox container \"ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b\" network for pod \"kafka-client-provisioner-5699f4779d-bhdmr\": networkPlugin cni failed to set up pod \"kafka-client-provisioner-5699f4779d-bhdmr_kafka\" network: failed to read MTU file: strconv.Atoi: parsing \"\": invalid syntax" pod="kafka/kafka-client-provisioner-5699f4779d-bhdmr"
5月 11 21:14:41 k8s kubelet[21929]: E0511 21:14:41.987224 21929 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kafka-client-provisioner-5699f4779d-bhdmr_kafka(ce19c239-a8df-43f4-a0ea-b90fc578cb28)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kafka-client-provisioner-5699f4779d-bhdmr_kafka(ce19c239-a8df-43f4-a0ea-b90fc578cb28)\\\": rpc error: code = Unknown desc = failed to set up sandbox container \\\"ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b\\\" network for pod \\\"kafka-client-provisioner-5699f4779d-bhdmr\\\": networkPlugin cni failed to set up pod \\\"kafka-client-provisioner-5699f4779d-bhdmr_kafka\\\" network: failed to read MTU file: strconv.Atoi: parsing \\\"\\\": invalid syntax\"" pod="kafka/kafka-client-provisioner-5699f4779d-bhdmr" podUID=ce19c239-a8df-43f4-a0ea-b90fc578cb28
5月 11 21:14:42 k8s kubelet[21929]: I0511 21:14:42.794079 21929 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="networkPlugin cni failed on the status hook for pod \"kafka-client-provisioner-5699f4779d-bhdmr_kafka\": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container \"ae93f2c9c0c7a0a2a96040c1ca38b390c7a4cd405b3781018722d84e5b0e2f9b\""
k8s v1.21.5 docker 20.10.8
重启过docker,kubelet, network,执行过docker system prune均不能恢复
该回答通过自己思路及引用到GPTᴼᴾᴱᴺᴬᴵ搜索,得到内容具体如下:
根据您提供的日志信息,可以看出是由于容器网络插件(CNI)无法设置POD的网络导致的问题。具体来说,CNI插件在尝试为POD设置网络时失败,因为它无法读取MTU文件并解析为无效的语法。这个错误可能是由于节点上的某些网络配置错误或损坏的文件引起的。
为了解决这个问题,您可以尝试以下几个步骤:
确认网络插件是否正确安装并正在运行。可以使用kubectl命令检查网络插件的状态。例如,对于Calico插件,您可以运行以下命令:
kubectl get pods -n kube-system -l k8s-app=calico-node
如果插件状态不是"Running"或"Completed",则需要修复插件的配置或重新安装它。
检查节点上的网络配置是否正确。您可以检查网络配置文件,如/etc/sysconfig/network-scripts/ifcfg-*以及网络服务配置文件,如/etc/sysconfig/network和/etc/resolv.conf。确保这些文件中的配置正确,并且没有损坏的文件或语法错误。
尝试重新安装CNI插件。如果CNI插件损坏或配置错误,您可以尝试重新安装它来修复问题。您可以通过以下命令重新安装Calico插件:
kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
如果您使用的是不同的插件,请使用相应的安装命令。
检查磁盘空间是否已释放。根据您提供的信息,节点上的磁盘空间已满,这可能导致其他问题。请确保您已释放足够的磁盘空间,以便Kubernetes可以正常工作。
如果以上步骤都没有解决问题,您可以考虑重新启动节点或重新安装Kubernetes集群来解决问题。
如果以上回答对您有所帮助,点击一下采纳该答案~谢谢