Service cluster ip changed but ingress-controller not update the ip

[mjhalxx]

What happened:

• set ingress use service

location /apis {
   proxy_pass http://gateway-proxy.gloo:80
}

k -n gloo get svc -o wide
gateway-proxy      ClusterIP  10.43.239.132  <node>   80/TCP 443/TCP

• helm uninstall gloo and install again, the servic cluster ip changed

k -n gloo get svc -o wide
gateway-proxy      ClusterIP  10.43.129.157  <node>   80/TCP 443/TCP

• ingress controller still use old service cluster ip and return 504

k -n network logs -f ingress-nginx-controoler-75dbd7bdd7-tr76b

[xxxx] "GET  /apis/xxxx"  504  xxxx  10.43.239.132:80 0 60.001 504

• restart ingress-nginx-controller pod will solve it

What you expected to happen:

• when service cluster ip change ，ingress-nginx should update the new ip， because the upstream is service name and the ip can get from dns

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

• Version is Release: v1.9.4

Kubernetes version (use kubectl version):

Environment:

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

@mjhalxx we dont test gloo.

I am using controller v1.9.4 and I use a simple pod using image nginx:alpine and this problem is not reproduced.

/remove-kind bug

you can try to post the answers to the questions asked in a new issue template. It will help to create some actionable comments.

/triage needs-information

[mjhalxx]

@longwuyuan You don’t need to use gloo. You can use any service. Instead of accessing the pod directly, you access the service. If you delete the service and then create a service with the same name, problems will arise. not restart pod but delete the service
and create again，so that the service clusterIP will change， but ingress-controller still use old service clusterIP

[mjhalxx]

@longwuyuan ingress-nginx usage scenarios are as follows，when srv ip change ingress-ngnix can not update：

[longwuyuan]

deleting a backend service and then recreating a replacement for that deleted service that was already configured in a ingress object is not a use-case that is tested in the CI.

It is better to use supported well planned workflow and use-cases for example you create a app and then you create a service for the app of type ClusterIP. Then you expose the app 's service with a ingress. Later, for any future maintenance of the app, just change image in the app pod

[mjhalxx]

@longwuyuan yes，but sometime backend service has some problem，we fix it，and reinstall it，then we found it can not reach again， and we have to restart ingress-nginx to resolve it

[longwuyuan]

that is not normal. most people change the image in the app. if you need to keep changing the service of type CLusterIP or other K8S objects to fix an app, then the problem is outside ingress controller.

what do you mean reinstall ? Do you delete the deployment/statefulset/daemonset ? and recreate it ?

[mjhalxx]

@longwuyuan we use gloo， it use helm to install and uninstall，then we upgrade gloo， it will reinstall it

[longwuyuan]

I don't know what gloo is but do you know if it deletes the service type ClusterIP ? If it does then this is bad design. You should find a way to only change the image in the pod and any other pod specs instead of deleting the service of type ClusterIP.

[bmv126]

@mjhalxx
How are you setting this:

location /apis {
proxy_pass http://gateway-proxy.gloo:80
}

Internally ingress-nginx uses lua, so if we create ingress resource, we typically see below rendered in nginx.conf

location /apis {
   ...
   ...
   proxy_pass http://upstream_balancer:80
}

So more information will help on how you are creating ingress resource.

[mjhalxx]

@bmv126 The actual configuration is as you said, but some parts are omitted and the main configuration is retained. What really confuses me is that the upstream here is the service domain name instead of the IP directly used. Why can't the service IP be updated in real time after it changes? , and it needs to be restarted before it can be updated.