Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm chart: upgrade with the same value (ipFamilyPolicy=RequireDualStack) not possible #11068

Closed
tibeer opened this issue Mar 5, 2024 · 13 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@tibeer
Copy link

tibeer commented Mar 5, 2024

What happened:

Tried to upgrade a helm-installation with the exact same values used during installation. Reproduced this with a new cluster and installation:

  1. use this command to install ingress-nginx:
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace --reuse-values --set resources=null --set controller.service.ipFamilyPolicy=RequireDualStack --wait
  1. Once everything is running, use the same command again, to do an "upgrade" and receive this error:
Error: UPGRADE FAILED: cannot patch "ingress-nginx-controller" with kind Service: Service "ingress-nginx-controller" is invalid: spec.ipFamilyPolicy: Invalid value: "RequireDualStack": must be 'SingleStack' to release the secondary IP family
  1. Important: This was never working before, so no degredation probably.

What you expected to happen:

The upgrade should do nothing, but just work.
One might wonder, why I want to do this: The command is part of an idempotent script that should always define the desired end status.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.0
  Build:         71f78d49f0a496c31d4c19f095469f3f23900f8a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.29.2
Server Version: v1.28.7

Environment:

  • Cloud provider or hardware configuration:

Oracle cloud ARM server, 4 cores 24GB RAM

  • OS (e.g. from /etc/os-release):
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
  • Kernel (e.g. uname -a):
Linux beta 5.15.0-1052-oracle #58-Ubuntu SMP Tue Feb 13 19:44:34 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
  • Install tools:
    • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

kubeadm

  • Basic cluster related info:
    • kubectl version

see above

  • kubectl get nodes -o wide
NAME   STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
beta   Ready    control-plane   136m   v1.28.2   10.0.0.113    <none>        Ubuntu 22.04.3 LTS   5.15.0-1052-oracle   containerd://1.7.2
  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress
ingress-nginx	ingress-nginx  	7       	2024-03-05 09:25:30.919411 +0100 CET	failed  	ingress-nginx-4.10.0   	1.10.0
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
USER-SUPPLIED VALUES:
controller:
  service:
    ipFamilyPolicy: RequireDualStack
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

N/A

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

N/A

  • Current State of the controller:
    • kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.0
              helm.sh/chart=ingress-nginx-4.10.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
NAME                                            READY   STATUS    RESTARTS   AGE   IP                NODE   NOMINATED NODE   READINESS GATES
pod/ingress-nginx-controller-6dc8c8fdf4-j2dbw   1/1     Running   0          64m   192.168.127.202   beta   <none>           <none>

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP                                          PORT(S)                      AGE   SELECTOR
service/ingress-nginx-controller             LoadBalancer   10.96.40.7      10.0.0.113,2603:c020:FOO:BAR:BAZ:XXXX:XXXX:XXXX  # removed by @tibeer for privacy reasons   80:32085/TCP,443:31820/TCP   64m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP      10.96.130.223   <none>                                               443/TCP                      64m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                                                                     SELECTOR
deployment.apps/ingress-nginx-controller   1/1     1            1           64m   controller   registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                                                                                     SELECTOR
replicaset.apps/ingress-nginx-controller-6dc8c8fdf4   1         1         1       64m   controller   registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6dc8c8fdf4
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
Name:             ingress-nginx-controller-6dc8c8fdf4-j2dbw
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             beta/10.0.0.113
Start Time:       Tue, 05 Mar 2024 08:33:55 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.10.0
                  helm.sh/chart=ingress-nginx-4.10.0
                  pod-template-hash=6dc8c8fdf4
Annotations:      cni.projectcalico.org/containerID: 4acc4ae5ba95163929ff2c3ea8c65746569da4aa7f20ae816f94e8ceccd4ccb3
                  cni.projectcalico.org/podIP: 192.168.127.202/32
                  cni.projectcalico.org/podIPs: 192.168.127.202/32,2001::8e4f:bc30:568f:1b49/128
Status:           Running
IP:               192.168.127.202
IPs:
  IP:           192.168.127.202
  IP:           2001::8e4f:bc30:568f:1b49
Controlled By:  ReplicaSet/ingress-nginx-controller-6dc8c8fdf4
Containers:
  controller:
    Container ID:    containerd://5397ec999cec78d787498432c11efa53e4574d601df7bf9888b03297d3fcb4dc
    Image:           registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-metrics=false
    State:          Running
      Started:      Tue, 05 Mar 2024 08:33:59 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6dc8c8fdf4-j2dbw (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-blvbj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-blvbj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.10.0
                          helm.sh/chart=ingress-nginx-4.10.0
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: ingress-nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         RequireDualStack
IP Families:              IPv4,IPv6
IP:                       10.96.40.7
IPs:                      10.96.40.7,2001:4::f54a
LoadBalancer Ingress:     10.0.0.113, 2603:c020:FOO:BAR:BAZ:XXXX:XXXX:XXXX  # removed by @tibeer for privacy reasons
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32085/TCP
Endpoints:                192.168.127.202:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31820/TCP
Endpoints:                192.168.127.202:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
  • Current state of ingress object, if applicable:
    • kubectl -n <appnamespace> get all,ing -o wide

N/A

  • kubectl -n <appnamespace> describe ing <ingressname>

N/A

  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

N/A

  • Others:
    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help

N/A

How to reproduce this issue:

Already described above :)

Anything else we need to know:

It might be also some misunderstanding on my side. Anyhow, I wasn't able to find anything similar in the issues here, so at least if someone else stumbles on this in the future, it might be helpful :)

@tibeer tibeer added the kind/bug Categorizes issue or PR as related to a bug. label Mar 5, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Mar 5, 2024
@longwuyuan
Copy link
Contributor

  • The project does not test upgrade dualstack on AWS so e could use more data
  • The message in error is about releasing and recreating
Service "ingress-nginx-controller" is invalid: spec.ipFamilyPolicy: Invalid value: "RequireDualStack": must be 'SingleStack' to release the secondary IP family

This hints that a very clear targeted test is needed to be done by a developer to triage this

@longwuyuan
Copy link
Contributor

/remove-kind bug
/triage needs-information
/kind feature

cc @Gacko

@k8s-ci-robot k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. kind/feature Categorizes issue or PR as related to a new feature. and removed kind/bug Categorizes issue or PR as related to a bug. labels Mar 5, 2024
@Gacko
Copy link
Member

Gacko commented Mar 5, 2024

That's not AWS, that's Oracle Cloud. I cannot test this, but I rather feel like that's an error produced by the according load balancer / cloud controller, so nothing caused by the chart or us.

@Gacko
Copy link
Member

Gacko commented Mar 5, 2024

@tibeer
Copy link
Author

tibeer commented Mar 5, 2024

To make things clear, I do not use oracle load balancers but metallb ^^

@Gacko
Copy link
Member

Gacko commented Mar 5, 2024

According to the code, this error message happens when you're handing in an ipFamilyPolicy other than SingleStack (that's the case) and the updated service has fewer clusterIPs and/or ipFamilies than the old service. Can you try omitting the ipFamilyPolicy from your values? As the service already exists, I don't think the assigned IPs are getting dropped. Anyway, please do so in a testing environment.

@tibeer
Copy link
Author

tibeer commented Mar 5, 2024

What's the deal then if the error comes from Kubernetes? Shouldn't the helm template render exactly the same values for the yaml files that get send to the kubernetes api? In this case, kubernetes shouldn't do anything but rather reply with "ok", desired state already reached.

To verify, I dumped the current service as a yaml file and tried to push it straight as it is again with "kubectl apply -f":

kubectl -n ingress-nginx apply -f svc.yaml
Error from server (Conflict): error when applying patch:
{"metadata":{"resourceVersion":"10892"}}
to:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "ingress-nginx-controller", Namespace: "ingress-nginx"
for: "svc.yaml": error when patching "svc.yaml": Operation cannot be fulfilled on services "ingress-nginx-controller": the object has been modified; please apply your changes to the latest version and try again

Does the nginx helm-chart admission controller permanently update the resource? Because I needed maybe 5 seconds between getting the yaml output and sending it again.

Or is it a missing kubernetes feature/bug? If yes, I can close this issue and raise one on kubernetes side :)

@Gacko
Copy link
Member

Gacko commented Mar 5, 2024

The Service resource is not checked or manipulated by the Ingress NGINX chart. We do not deploy any webhooks for Service resources. Also you probably cannot kubectl apply those resources if they have been created by doing so. Again, that's nothing specific to Ingress NGINX, that's a Kubernetes in general topic.

@tibeer
Copy link
Author

tibeer commented Mar 5, 2024

Thanks! I will try to ask why this behaviour is desired on the kubernetes side :)

@tibeer
Copy link
Author

tibeer commented Mar 8, 2024

@Gacko I will reopen this issue, as I am pretty sure, that the error message is misleading, because I ran helm with --dry-run=server to see the actual patch templates:

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.10.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: "1.10.0"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ipFamilyPolicy: RequireDualStack
  ipFamilies: 
    - IPv4
  ports:
  (...)

The problem might be, that the ipFamiles does not contain IPv6, therefore kuberentes detechts a change and runs into an issue that it does not allow RequireDualStack in combination with only one ipFamily set.
I am not 100% sure, but I also mentioned this on the kubernetes ticket itself: kubernetes/kubernetes#123761
And I am unsure, who's behaviour here is not working as expected. Might even be helm itself 🤔

@tibeer tibeer reopened this Mar 8, 2024
@Gacko
Copy link
Member

Gacko commented Mar 8, 2024

The check in the code compares the old ipFamilies to the new ones and complains if you're reducing them, so from IPv4 & IPv6 to only IPv4 e.g., while still using RequireDualStack.

The Ingress NGINX chart has a property controller.service.ipFamilies which defaults to just IPv4. So if you change controller.service.ipFamilyPolicy while not changing controller.service.ipFamilies, you might run into that issue.

Unfortunately Kubernetes is maybe not complaining about that on creation and just adds IPv6 after creation to comply to your ipFamilyPolicy, but without changing ipFamilies to a matching value, it's quite clear why it is no longer working on updates.

Sorry that I didn't figure that earlier! I guess you're expecting to just set controller.service.ipFamilyPolicy and not care about controller.service.ipFamilies, right? I'm not sure if we can really have a good default behavior here as other users might eventually want to default to ipFamilyPolicy SingleStack with ipFamilies IPv6.

Of course setting ipFamilyPolicy to RequireDualStack might make one think "yeah, just set IPv4 and IPv6 in ipFamilies then, I don't wanna care", but actually order matters here. Kubernetes is acting different depending on what's mentioned first in the ipFamilies.

So in the end I don't think there is a one-fits-all default behavior we can implement and one always needs to configure both, ipFamilyPolicy and ipFamilies.

cc @strongjz @rikatz in case you are more into the whole "Kubernetes on Dual Stack" topic. 🙂

@tibeer
Copy link
Author

tibeer commented Mar 8, 2024

Just tested it. Indeed, setting the ipFamilies resolves my problem. I guess there is no need to catch this edge case, since it was a user error. But at least it is now documented here in an issue so others can find it 😄 Thanks a lot for your help!

@tibeer tibeer closed this as completed Mar 8, 2024
C0nsultant added a commit to C0nsultant/homelab-as-code that referenced this issue Dec 17, 2024
avoid kubernetes/ingress-nginx#11068
the chart implicitly assumes IPv4 only, even when RequireDualStack
C0nsultant added a commit to C0nsultant/homelab-as-code that referenced this issue Dec 17, 2024
avoid kubernetes/ingress-nginx#11068
the chart implicitly assumes IPv4 only, even when RequireDualStack
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
Development

No branches or pull requests

4 participants