100% CPU usage of nginx worker process #10992
Comments
k8s-ci-robot
added
the
needs-triage
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Just found out that it could be related to this bug https://trac.nginx.org/nginx/ticket/2548. As you can see in the list of annotations I have there used annotation Currently I removed this annotation from that ingress resource and I will wait to see if the CPU will be on 100% again or not. |
|
/remove-kind bug
/triage needs-information |
k8s-ci-robot
added
triage/needs-information
|
and yes, this link says default value is 8K https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-body-buffer-size and so setting a value of 0 by yourself and then reporting a issue here is very odd |
|
I would not say that it is very odd. Becuase if you have one big cluster where every developer can deploy their ingresses with ther own configuration and just one wrong configuration could take down the whole cluster is not so good. Have you ever thought about validating values of annotations with a webhook or something similar? Because this is the good candidate to validate that value is either 8k or 16k, everything else is denied as it could cause unexpected problems. Neverthless the CPU is still at normal value so removing annotation |
|
can you close the issue if resolved. if you do not allow any bits/bytes in client body buffer then there will be a infinite loop until connection is terminated. so while your interest to protect seems valid, it seems like a impractical config to deploy. but this is just my guess. so please close the issue if resolved |
it doesnt seem to be proper nginx behaviour, since local (un-kubernetes) nginx behaves differently on 0 as 'allow all' |
What happened:
We are using on-premise K8s via TKGI from VMware. When we upgraded from TKGI 1.16.3 (Kubernetes v1.25.12) to 1.18.1 (Kubernetes v1.27.8), after a few days there was a problem that ingress-nginx pods were randomly mining all CPU cores at 100%. First it happened with one pod, then with another, and so gradually until ingress-nginx was using 100% of the entire cluster's CPU. The only temporary solution is to kill these pods to create new ones that already have a good CPU.
What you expected to happen:
CPU of ingress-nginx remains at normal – typically for our environment on ~10-20m CPU instead of 1000,2000 or 3000m CPU.
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Kubernetes version (use
kubectl version):v1.27.8
Environment:
Cloud provider or hardware configuration: VMware TKGI
OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
Kernel (e.g.
uname -a): Linux d95608af-3eb5-4d69-ad64-5603722db030 6.2.0-39-generic Developer documentation #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/LinuxInstall tools:
Basic cluster related info:
kubectl versionClient Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8+vmware.1", GitCommit:"dba4dc9cc64d96ee4d003860d8fdb1722db28eb0", GitTreeState:"clean", BuildDate:"2023-11-17T05:55:19Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}kubectl get nodes -o wideHow was the ingress-nginx-controller installed:
Cluster has two instance of ingress-nginx installed via Helm using these values
Values of first instance
Values of second instance
Current State of the controller:
kubectl describe ingressclassesName: foo Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=foo app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.5 helm.sh/chart=ingress-nginx-4.9.0 Annotations: <none> Controller: example.com/foo Events: <none> Name: bar Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=bar app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.5 helm.sh/chart=ingress-nginx-4.9.0 Annotations: <none> Controller: example.com/bar Events: <none>kubectl -n <ingresscontrollernamespace> get all -A -o widekubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>Name: ingress-nginx-controller-84bd457d8c-z5cvl Namespace: ingress-nginx Priority: 0 Node: 3b4008ad-2a25-4eb0-9f2d-a418381e3c99/10.141.0.15 Start Time: Sat, 17 Feb 2024 21:41:39 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=foo app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.5 helm.sh/chart=ingress-nginx-4.9.0 pod-template-hash=84bd457d8c Annotations: kubectl.kubernetes.io/restartedAt: 2024-02-17T20:41:39Z Status: Running IP: X.X.X.22 IPs: IP: X.X.X.22 Controlled By: ReplicaSet/ingress-nginx-controller-84bd457d8c Containers: controller: Container ID: containerd://a2c0dea5995e6a26f46de6d510ea85d44c4d0068dc865753d3e4b64594842e08 Image: registry.k8s.io/ingress-nginx/controller:v1.9.5@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e Image ID: registry.k8s.io/ingress-nginx/controller@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=example.com/foo-access --ingress-class=foo --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key --ingress-class-by-name=true --enable-ssl-passthrough --v=5 State: Running Started: Sat, 17 Feb 2024 21:41:43 +0100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 300Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-84bd457d8c-z5cvl (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qkr78 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-qkr78: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=internal app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.9.5 argocd.argoproj.io/instance=k8s-test-01-ingress-nginx helm.sh/chart=ingress-nginx-4.9.0 Annotations: ncp/internal_ip_for_policy: X.X.X.X Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=internal,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: X.X.X.X IPs: X.X.X.X IP: X.X.X.X LoadBalancer Ingress: X.X.X.X Port: http 80/TCP TargetPort: http/TCP NodePort: http 31139/TCP Endpoints: X.X.X.11:80,X.X.X.12:80,X.X.X.13:80 + 17 more... Port: https 443/TCP TargetPort: https/TCP NodePort: https 31349/TCP Endpoints: X.X.X.11:443,X.X.X.12:443,X.X.X.13:443 + 17 more... Session Affinity: None External Traffic Policy: Cluster Events: <none>Current state of ingress object, if applicable:
Others:
I enabled debug mode
--v=5and found out that when CPU of the nginx worker process is on 100 % then in the logs of this pod isFrom this log I found out that this debug msg is from process with PID 712 which uses so much CPU. The process is probably stuck in some loop. I found out that this msg comes from this line https://github.com/nginx/nginx/blob/97a111c0c0a40ecaa7771ecec66b8ed37b0350d5/src/http/ngx_http_request_body.c#L402 which is part of func
ngx_http_do_read_client_request_body. So maybe it is related to the we have setproxy-body-size: "0"for ingress-nginx, or maybe some ingress's annotation could cause this issue??How to reproduce this issue:
Hard to reproduce this issue, even in our environment we couldn't do that. It happens randomly.
The text was updated successfully, but these errors were encountered: