canary doesn't work #11085

Yelijah · 2024-03-11T06:09:49Z

What happened:
I create s simple ingress example with canary annotations，but when i visit my url，it returns 404 error page.

my ingress is like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-canary-mysqld-exporter
  namespace: dev
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "region"
    nginx.ingress.kubernetes.io/canary-by-header-value: "mysqld"
spec:
  rules:
  - http:
      paths:
      - path: /metrics
        pathType: Prefix
        backend:
          service:
            name: mysqld-exporter-prometheus-mysql-exporter
            port:
              number: 9104
    host: www.host3.com
  ingressClassName: nginx

My visit is like:

$  curl -H "Host: www.host3.com" -H "region: mysqld" http://www.host3.com:32080/metrics
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

if I delete annotation about canary, my ingress work well. so it can't be my backend's wrong.

What you expected to happen:
I hope my canary can work well

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.8.5
Build: b5595e1
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:15:38Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): centos7
Kernel (e.g. uname -a): 3.10.0-957.el7.x86_64
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
Basic cluster related info:
- kubectl version
- kubectl get nodes -o wide
How was the ingress-nginx-controller installed:

$ helm ls -n dev|grep ingress
ingress-nginx   dev             1               2024-03-11 12:46:13.634910245 +0800 CST deployed        ingress-nginx-4.7.5             1.8.5

$ helm get values ingress-nginx -n dev -o yaml
controller:
  admissionWebhooks:
    enabled: false
  allowSnippetAnnotations: true
  config:
    client-header-buffer-size: 512k
    large-client-header-buffersl: 4 512k
    proxy-body-size: 100M
    proxy-buffer-size: 64k
    proxy-buffers-number: 32
    proxy-connect-timeout: 75
    proxy-read-timeout: 400
    proxy_send_timeout: 400
  enableAnnotationValidations: false
  extraArgs:
    metrics-per-host: false
  image:
    chroot: false
    digest: null
    digestChroot: null
    image: ingress-nginx/controller
    registry: registry.k8s.io
    tag: v1.8.5
  metrics:
    enabled: true
  name: controller
  service:
    nodePorts:
      http: "32080"
      https: "32443"
      tcp: {}
      udp: {}
    type: NodePort

Current State of the controller:

$ kubectl get all -o wide -n dev|grep ingress
pod/ingress-nginx-controller-7688b96c69-smwj5                   1/1     Running   0          77m   10.244.123.185   k8s.node2.fanruan   <none>           <none>

service/ingress-nginx-controller                    NodePort    10.104.90.205    <none>        80:32080/TCP,443:32443/TCP   77m    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

service/ingress-nginx-controller-metrics            ClusterIP   10.101.108.85    <none>        10254/TCP                    77m    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

deployment.apps/ingress-nginx-controller                    1/1     1            1           77m   controller                  docker.io/yelijah/ingress-nginx-controller:v1.8.5   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

replicaset.apps/ingress-nginx-controller-7688b96c69                   1         1         1       77m   controller                  docker.io/yelijah/ingress-nginx-controller:v1.8.5   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7688b96c69

Current state of ingress object, if applicable:

$ kubectl get ing -o wide -n dev
NAME                             CLASS   HOSTS           ADDRESS         PORTS   AGE
ingress-canary-mysqld-exporter   nginx   www.host3.com   10.104.90.205   80      72m

$ kubectl describe ing ingress-canary-mysqld-exporter -n dev
Name:             ingress-canary-mysqld-exporter
Labels:           <none>
Namespace:        dev
Address:          10.104.90.205
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.host3.com  
                 /metrics   mysqld-exporter-prometheus-mysql-exporter:9104 (10.244.48.234:9104)
Annotations:     nginx.ingress.kubernetes.io/canary: true
                 nginx.ingress.kubernetes.io/canary-by-header: region
                 nginx.ingress.kubernetes.io/canary-by-header-value: mysqld
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    12m (x5 over 72m)  nginx-ingress-controller  Scheduled for sync

$  curl -H "Host: www.host3.com" -H "region: mysqld" http://www.host3.com:32080/metrics
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

Others:
- Any other related information like ;
  - copy/paste of the snippet (if applicable)
  - kubectl describe ... of any custom configmap(s) created and in use
  - Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2024-03-11T06:09:57Z

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

longwuyuan · 2024-03-11T07:13:39Z

/remove-kind bug
/triage needs-information
/kind support

I can not reproduce any problem with this example https://kubernetes.github.io/ingress-nginx/examples/canary/

Yelijah · 2024-03-11T07:43:10Z

/remove-kind bug /triage needs-information /kind support

I can not reproduce any problem with this example https://kubernetes.github.io/ingress-nginx/examples/canary/

the example uses 'nginx.ingress.kubernetes.io/canary-weight', but my ingress uses 'nginx.ingress.kubernetes.io/canary-header'

longwuyuan · 2024-03-11T08:28:38Z

show your data that is proof of the problem
here is my data that I can not reproduce

Yelijah · 2024-03-11T08:37:33Z

show your data that is proof of the problem

here is my data that I can not reproduce

I have found the problem, my canary ingress hasn't the host; but my main ingress has the host.
It seems the canary ingress and the main ingress must has the same host.

longwuyuan · 2024-03-11T08:39:00Z

close the issue

longwuyuan · 2024-03-11T08:42:14Z

/close

as issue resolved

k8s-ci-robot · 2024-03-11T08:42:18Z

@longwuyuan: Closing this issue.

In response to this:

/close

as issue resolved

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Yelijah added the kind/bug Categorizes issue or PR as related to a bug. label Mar 11, 2024

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Mar 11, 2024

k8s-ci-robot added the needs-priority label Mar 11, 2024

strongjz added this to [SIG Network] Ingress NGINX Mar 11, 2024

k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Mar 11, 2024

k8s-ci-robot closed this as completed Mar 11, 2024

github-project-automation bot moved this to Done in [SIG Network] Ingress NGINX Mar 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

canary doesn't work #11085

canary doesn't work #11085

Yelijah commented Mar 11, 2024

k8s-ci-robot commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

Yelijah commented Mar 11, 2024 •

edited

Loading

longwuyuan commented Mar 11, 2024

Yelijah commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

k8s-ci-robot commented Mar 11, 2024

canary doesn't work #11085

canary doesn't work #11085

Comments

Yelijah commented Mar 11, 2024

k8s-ci-robot commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

Yelijah commented Mar 11, 2024 • edited Loading

longwuyuan commented Mar 11, 2024

Yelijah commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

longwuyuan commented Mar 11, 2024

k8s-ci-robot commented Mar 11, 2024

Yelijah commented Mar 11, 2024 •

edited

Loading