SSL alert number 40 when updating from 1.8.x to 1.9.x

[JonasJes]

What happened:

When updating from 1.8.1 to 1.9.5 through the helm chart we experienced the following error on our only ExternalName service

[error] 96#96: *5502 SSL_do_handshake() failed (SSL: error:0A000410:SSL routines::sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: XXX.XXX.XXX.XXX, server: www.xxx.com, request: "POST /api/cms/navigation/context/user HTTP/2.0", upstream: "https://XXX.XXX.XXX.XXX:443/api/cms/navigation/context/user", host: "www.xxx.com"

I tried downgrading to 1.9.1 (Helm 4.8.1)** was also not working.
Then downgrading to 1.8.2 (Heml 4.7.2) which was working.

All the other services that didn't use ExternalName was working

What you expected to happen:

The SSL handshake is working for ExternalName services as in pre-1.8.x

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v1.9.5

Kubernetes version (use kubectl version):
v1.27.3

Environment:

• Cloud provider or hardware configuration: AKS
• OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
• Kernel (e.g. uname -a): 5.15.0-1049-azure
• Install tools: Azure CLI
• Basic cluster related info:

Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"31eb725f9812713ec7fd6cf293466ce4fdc7efa8", GitTreeState:"clean", BuildDate:"2023-10-09T14:42:41Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

NAME                                 STATUS   ROLES   AGE   VERSION   INTERNAL-IP        EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-38934212-vmss000000    Ready    agent   92d   v1.27.3   XXX.XXX.XXX.XXX    <none>        Ubuntu 22.04.3 LTS   5.15.0-1053-azure   containerd://1.7.5-1

• How was the ingress-nginx-controller installed:
  Using Helm ingress-nginx-4.9.0

NAME             NAMESPACE  REVISION  UPDATED                                 STATUS     CHART                 APP VERSION
staging-ingress  staging    31        2024-01-19 10:50:59.737488184 +0000 UTC deployed   ingress-nginx-4.9.0   1.9.5

• Output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
              - ingress-nginx
            - key: app.kubernetes.io/instance
              operator: In
              values:
              - staging-ingress
            - key: app.kubernetes.io/component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
        weight: 100
  config:
    force-ssl-redirect: "true"
    large-client-header-buffers: 4 32k
    proxy-body-size: 20m
    ssl-redirect: "true"
  ingressClassResource:
    name: staging-ingress
  minAvailable: 1
  replicaCount: 2
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    loadBalancerIP: XXX.XXX.XXX.XXX
  tolerations:
  - effect: NoSchedule
    key: namespace
    operator: Equal
    value: staging
defaultBackend:
  enabled: true
  tolerations:
  - effect: NoSchedule
    key: namespace
    operator: Equal
    value: staging

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         staging-ingress
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=staging-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.9.5
              helm.sh/chart=ingress-nginx-4.9.0
Annotations:  meta.helm.sh/release-name: staging-ingress
              meta.helm.sh/release-namespace: staging
Controller:   k8s.io/ingress-nginx
Events:       <none>

• Others:
  Values.yml for helm

controller:
  config: 
    proxy-body-size: "20m"
    large-client-header-buffers: "4 32k"
    force-ssl-redirect: "true"
    ssl-redirect: "true"
  ingressClassResource:
    name: staging-ingress
  tolerations:
    - key: "namespace"
      operator: "Equal"
      value: "staging"
      effect: "NoSchedule"
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
              - ingress-nginx
            - key: app.kubernetes.io/instance
              operator: In
              values:
              - staging-ingress
            - key: app.kubernetes.io/component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  replicaCount: 2
  minAvailable: 1
  service:
    loadBalancerIP: "XXX.XXX.XXX.XXX"
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
defaultBackend:
  enabled: true
  tolerations:
    - key: "namespace"
      operator: "Equal"
      value: "staging"
      effect: "NoSchedule"

Ingress.yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: epi-cms-content
  labels:
    app: epi-cms-content
    tier: backend
    app.kubernetes.io/component: webapi
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/upstream-vhost: xxx.xxx.com
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,recaptcha,X-Robots-Tag
    nginx.ingress.kubernetes.io/cors-expose-headers: X-XXX-Redirect
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_name xxx.xxx.com;
      proxy_ssl_server_name on;
      proxy_set_header User-Agent EPI-Proxy;
spec:
  tls:
  - hosts:
    - www.xxx.com
    secretName: tls-secret
  rules:
  - host: www.xxx.com
    http:
      paths:
      - backend:
          service:
            name: epi-cms
            port:
              number: 443
        path: /content(/|$)(.*)
        pathType: ImplementationSpecific
  ingressClassName: staging-ingress

Service.yml:

kind: Service
apiVersion: v1
metadata:
  name: epi-cms
  labels:
    app: epi-cms
    tier: backend
    apiarea: cms
    app.kubernetes.io/component: webapi
spec:
  type: ExternalName
  externalName: xxx.xxx.com
  ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80

How to reproduce this issue:

1. Create AKS cluster
2. Create namespace staging
3. Cange domains and IP addresses in values.yml, ingress.yml, service.yml
4. Install ingress-nginx with values from above

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --namespace staging --install --values values.yml --wait staging-ingress ingress-nginx/ingress-nginx

5. Apply Service.yml and ingress.yml to the cluster
6. Test any endpoint
7. View logs from the Ingress Controller

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[strongjz]

Another aks issue granted it's also ssl proxying to external @Gacko

[strongjz]

Does the proxy_ssl_name need to match host header at host: www.xxx.com

I don't think we made any updates to SSL in 1.9.X right @rikatz

[Gacko]

According to my quick research, something with SNI seems to be wrong. So as @strongjz already mentioned, either the proxy_ssl_name (user defined server name) or proxy_ssl_server_name (use server's name, no manual setting) is wrong.

As you're setting the both of them, I do not know which one takes precedence. Have you tried using the built-in annotations instead of using a snippet?

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-certificate-authentication

Okay... sorry... I spent 15min on research just to find the real root cause: Please enable controller.allowSnippetAnnotations. 😅

https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.5/charts/ingress-nginx/values.yaml#L90

[Gacko]

You should have got some errors in your controller's logs.

[Gacko]

/assign

[JonasJes]

Hi @Gacko

Yes, that was it, setting controller.allowSnippetAnnotations: true fixed the problem, Thank You.
I will look into whether we can use the built-in annotations as I do not think they were not available when we created the ingress 4-5 years ago.

But I do agree with most of the comments from #10393 that the breaking change could have been better documented.

Isn't there anywhere we can see breaking changes except going through all the changelogs?

[strongjz]

@JonasJes is open to other options; we discuss the changes in our community meetings every other Thursday, and we have a Google group folks can subscribe to where we announce releases and another discussions on the Kubernetes slack channel ingress-nginx-dev.

We opened two release branches so folks had time to digest the new changes when moving from one minor to another; we apologize this wasn't clear in the error logs or elsewhere. We also released a beta version of this for testing before releasing an official 1.9 release.

I'm glad we found the fix; closing this issue and open a separate one for adding more logging in the controller logs. #10901

/close

[k8s-ci-robot]

@strongjz: Closing this issue.

In response to this:

@JonasJes is open to other options; we discuss the changes in our community meetings every other Thursday, and we have a Google group folks can subscribe to where we announce releases and another discussions on the Kubernetes slack channel ingress-nginx-dev.

We opened two release branches so folks had time to digest the new changes when moving from one minor to another; we apologize this wasn't clear in the error logs or elsewhere. We also released a beta version of this for testing before releasing an official 1.9 release.

I'm glad we found the fix; closing this issue and open a separate one for adding more logging in the controller logs. #10901

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JonasJes]

Hi @strongjz

The logging would be great.

But as a developer when I discover problems with newer versions, I usually go to the Release Notes as in https://github.com/kubernetes/ingress-nginx/releases

When #10393 already has the Breaking change (fix or feature that would cause existing functionality to change) checked, isn't there a way to get that under a section by itself in the release notes, to make it easier visible?

[strongjz]

The release notes and the changelog are the same for the most part; in 1.9.0, we added that testing should be done in the github release; naming the header breaking changes would have got more people's attention, I suppose.

https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.9.0