X-Forwarded-Port is always fixed to 443.

[jclab-joseph]

What happened:

X-Forwarded-Port is always fixed to 443.
Even if change --https-port, it is always fixed to 443.

Because of this, we got the following results:

GET /hello HTTP/1.1
Host: 172.17.0.4:5440
X-Request-ID: fa76d3f6f9e040a4e4efef1f78bc388a
X-Real-IP: 172.16.1.81
X-Forwarded-For: 172.16.1.81
X-Forwarded-Host: 172.17.0.4:5440
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Scheme: https
Origin: https://172.17.0.4:5440
...

In spring-boot, 403 is caused if the origin port and X-Forwarded-Port are different.

What you expected to happen:

It should be changed to https listen port.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
  Release:       1.10.0
  Build:         71f78d4
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

Kubernetes version (use kubectl version): v1.27.11+rke2r1

How to reproduce this issue:

helm install nginx-ingress bitnami/nginx-ingress-controller -f values.yaml

values.yaml:

ingressClassResource:
  name: test
  controllerClass: "k8s.io/ingress-nginx-test"
electionID: test-ingress-controller-leader
kind: DaemonSet
daemonset:
  useHostPort: true
  hostPorts:
    http: 5000
    https: 5443
containerPorts:
  http: 5000
  https: 5443
extraArgs:
  http-port: "5000"
  https-port: "5443"

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug
/triage needs-information

This is a ingress-controller, that processes ingress rules, that almost always contain a TLS section in real-world use case, and nobody wants their users to type a port number in browser or force suffix a port number in api calls. That is stating the obvious that port 443 is accepted standard.

To present the real info to the backend pod, headers like X-Forwarded-Proto and X-Forwarded-Port in addition to X-Forward-For etc contain the truth. So this issue description can e improved to contain more data than currently provided, to justify label it as a bug.

Several apps like Tomcat or others listen on 8080 or 8443 etc. But TLS config for them is a option and not defaulted out of the box. Hence the out of the box use case of ingress is termination on the the controller for TLS and backend plaintext HTTP communication to app. And then some people go around this by configuring SSL-Passthrough so TLS termination is on the pod. Hence detailed descriptive data and logic provided in this issue description will help a reader, the specs & design based on which the current behaviour is a bug. Hope you provide that.

The new bug report contains questions in the template. Answer them as that creates action items instead of reading this kind of vague content.

[jclab-joseph]

Currently, X-Forwarded-Port is transmitting lie information, not truth information. I still don't understand why this isn't a bug.
I think X-Forwarded-Port should tell the truth.
Although the https 443 port is common, I don't think there is any reason to force it.

This causes cors rejection in Spring framework.

[longwuyuan]

Your values file yaml hints that you are using hostPorts and you have ignored all the questions that are asked in the template of a new bug report.

So you can yourself set the bug tag if you want to but you are not really helping a reader fully understand the small tiny intricate critical details.

For example there is no data on the impact of hostPorts when compared to a normal service of --type LoadBalancer being the termination. And on how all this impacts kube-proxy etc.

[grigoni]

@jclab-joseph
searching in this repo lead me to following piece of code:

• ingress-nginx/rootfs/etc/nginx/template/nginx.tmpl

  Line 1439 in c25b80c

  {{ $proxySetHeader }} X-Forwarded-Port $pass_port;

• ingress-nginx/rootfs/etc/nginx/lua/lua_ingress.lua

  Line 139 in c25b80c

  ngx.var.pass_port = ngx.var.pass_server_port

{{ $proxySetHeader }} X-Forwarded-Port $pass_port;

ngx.var.pass_port = ngx.var.pass_server_port if config.is_ssl_passthrough_enabled then if ngx.var.pass_server_port == config.listen_ports.ssl_proxy then ngx.var.pass_port = 443 end elseif ngx.var.pass_server_port == config.listen_ports.https then ngx.var.pass_port = 443 end

could be related to your case?

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[jclab-joseph]

@grigoni That's probably right.
It doesn't seem like a good way, but I solved this problem by creating a plugin using a lua script.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-custom-lua
  namespace: kube-system
data:
  main.lua: |
    local ngx = ngx

    local _M = {}

    function _M.rewrite()
      ngx.var.pass_port = 5440
      ngx.var.pass_access_scheme = "https"
    end

    return _M

[longwuyuan]

The basic issue description is false and does not match a real practical curl test as seen below

% curl httpbun.dev.enjoydevops.com/headers
{
  "Accept": "*/*",
  "Host": "httpbun.dev.enjoydevops.com",
  "User-Agent": "curl/8.6.0",
  "X-Forwarded-For": "172.19.0.1",
  "X-Forwarded-Host": "httpbun.dev.enjoydevops.com",
  "X-Forwarded-Port": "80",
  "X-Forwarded-Proto": "http",
  "X-Forwarded-Scheme": "http",
  "X-Real-Ip": "172.19.0.1",
  "X-Request-Id": "a79b61e6b5fe3ea9963002dce5c545cf",
  "X-Scheme": "http"
}
[~] 
% k -n httpbun describe ing httpbun 
Name:             httpbun
Labels:           <none>
Namespace:        httpbun
Address:          172.19.0.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  httpbun.dev.enjoydevops.com  
                               /   httpbun:80 (10.244.0.2:80)
Annotations:                   nginx.ingress.kubernetes.io/force-ssl-redirect: false
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    38s (x3 over 23h)  nginx-ingress-controller  Scheduled for sync
[~] 
% 

Since there is no traction on this, I will close the issue for now.

Please re-open once there is data posted here or the issue description has improved to show the proof of a problem iwth the controller.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

The basic issue description is false and does not match a real practical curl test as seen below

% curl httpbun.dev.enjoydevops.com/headers
{
 "Accept": "*/*",
 "Host": "httpbun.dev.enjoydevops.com",
 "User-Agent": "curl/8.6.0",
 "X-Forwarded-For": "172.19.0.1",
 "X-Forwarded-Host": "httpbun.dev.enjoydevops.com",
 "X-Forwarded-Port": "80",
 "X-Forwarded-Proto": "http",
 "X-Forwarded-Scheme": "http",
 "X-Real-Ip": "172.19.0.1",
 "X-Request-Id": "a79b61e6b5fe3ea9963002dce5c545cf",
 "X-Scheme": "http"
}
[~] 
% k -n httpbun describe ing httpbun 
Name:             httpbun
Labels:           <none>
Namespace:        httpbun
Address:          172.19.0.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
 Host                         Path  Backends
 ----                         ----  --------
 httpbun.dev.enjoydevops.com  
                              /   httpbun:80 (10.244.0.2:80)
Annotations:                   nginx.ingress.kubernetes.io/force-ssl-redirect: false
Events:
 Type    Reason  Age                From                      Message
 ----    ------  ----               ----                      -------
 Normal  Sync    38s (x3 over 23h)  nginx-ingress-controller  Scheduled for sync
[~] 
% 

Since there is no traction on this, I will close the issue for now.

Please re-open once there is data posted here or the issue description has improved to show the proof of a problem iwth the controller.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aescaler-raft]

The basic issue description is false and does not match a real practical curl test as seen below

% curl httpbun.dev.enjoydevops.com/headers
{
  "Accept": "*/*",
  "Host": "httpbun.dev.enjoydevops.com",
  "User-Agent": "curl/8.6.0",
  "X-Forwarded-For": "172.19.0.1",
  "X-Forwarded-Host": "httpbun.dev.enjoydevops.com",
  "X-Forwarded-Port": "80",
  "X-Forwarded-Proto": "http",
  "X-Forwarded-Scheme": "http",
  "X-Real-Ip": "172.19.0.1",
  "X-Request-Id": "a79b61e6b5fe3ea9963002dce5c545cf",
  "X-Scheme": "http"
}
[~] 
% k -n httpbun describe ing httpbun 
Name:             httpbun
Labels:           <none>
Namespace:        httpbun
Address:          172.19.0.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  httpbun.dev.enjoydevops.com  
                               /   httpbun:80 (10.244.0.2:80)
Annotations:                   nginx.ingress.kubernetes.io/force-ssl-redirect: false
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    38s (x3 over 23h)  nginx-ingress-controller  Scheduled for sync
[~] 
% 

Since there is no traction on this, I will close the issue for now.

Please re-open once there is data posted here or the issue description has improved to show the proof of a problem iwth the controller.

/close

Your test clearly used HTTP when the original issue demonstrated this behavior over HTTPS.

The expected behavior is that ingress-nginx would properly set the value for "X-Forwarded-Port" in the header based on the port specified in the original request.

This is undocumented and bewildering behavior.

test instructions:

1. Deploy the ingress-nginx controller via the helm chart with --set controller.service.type=NodePort
2. Get the port number from the ingress-nginx-controller service and grab the IP address of one of the nodes
3. Create an ingress for your httpbun service with the requisite host and tls entries
4. Create an entry in /etc/hosts to match ingress's host and the node's IP address
5. Send a request to the ingress's host
6. Observe the wrong port in X-Forwarded-Port

[longwuyuan]

@aescaler-raft I will reopen and run test with HTTPS.
/reopen

Please confirm the below info for the test ;

• The installation has to be a daemonset
• The controller must useHostPort
  -The default controller --https-port must be changed from 443 to 5443

[k8s-ci-robot]

@longwuyuan: Reopened this issue.

In response to this:

@aescaler-raft I will reopen and run test with HTTPS.
/reopen

Please confirm the below info for the test ;

• The installation has to be a daemonset
• The controller must useHostPort
  -The default controller --https-port must be changed from 443 to 5443

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

@aescaler-raft also help out in the following way.

• Please do a quick test on a kind cluster that shows the wrong header
• Please post the values file you used for installing the controller on the kind cluster
• Please post the complete and actual curl request sent with -v
• If possible please comment on use of the httpbun image as workload for test
  • kubectl create ns httpbun
  • kubectl -n httpbun create deploy httpbun --image ghcr.io/sharat87/httpbun --port 80
  • kubectl -n httpbun expose deploy httpbun --port 80
  • kubectl -n httpbun create ingress httpbun --class nginx --rule httpbun.local/"*"=httpbun:80,tls

[longwuyuan]

This is what I did while waiting for a values file from you (for kind cluster install of controller) and the curl ;

% helm -n ingress-nginx get values ingress-nginx 
USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
controller:
  config:
    use-forwarded-headers: true
  extraArgs:
    default-ssl-certificate: ingress-nginx/wildcard.dev.enjoydevops.com
  metrics:
    enabled: true
    serviceMonitor:
      additionalLabels:
        release: prometheusgrafana
      enabled: true
  service:
    externalTrafficPolicy: Local
[~] 
% minikube ip
192.168.49.2
[~] 
% k -n httpbun describe ing httpbun 
Name:             httpbun
Labels:           <none>
Namespace:        httpbun
Address:          192.168.49.2
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes httpbun.dev.enjoydevops.com
Rules:
  Host                         Path  Backends
  ----                         ----  --------
  httpbun.dev.enjoydevops.com  
                               /   httpbun:80 (10.244.0.14:80)
Annotations:                   <none>
Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  Sync    25m    nginx-ingress-controller  Scheduled for sync
  Normal  Sync    21m    nginx-ingress-controller  Scheduled for sync
  Normal  Sync    9m27s  nginx-ingress-controller  Scheduled for sync
[~] 
% % curl -L `minikube ip`/headers -H "Host: httpbun.dev.enjoydevops.com"  --resolve httpbun.dev.enjoydevops.com:443:`minikube ip` -i
HTTP/1.1 308 Permanent Redirect
Date: Sat, 14 Sep 2024 12:44:08 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://httpbun.dev.enjoydevops.com/headers

HTTP/2 200 
date: Sat, 14 Sep 2024 12:44:08 GMT
content-type: application/json
content-length: 386
x-powered-by: httpbun/af040d24038613575a85f74c2283ae79f8169927
strict-transport-security: max-age=31536000; includeSubDomains

{
  "Accept": "*/*",
  "Host": "httpbun.dev.enjoydevops.com",
  "User-Agent": "curl/7.81.0",
  "X-Forwarded-For": "192.168.49.1",
  "X-Forwarded-Host": "httpbun.dev.enjoydevops.com",
  "X-Forwarded-Port": "443",
  "X-Forwarded-Proto": "https",
  "X-Forwarded-Scheme": "https",
  "X-Real-Ip": "192.168.49.1",
  "X-Request-Id": "3b72364e847b61e9b7113d20267f283f",
  "X-Scheme": "https"
}
[~] 

[pdefreitas]

Lets say you want to do some hardening and to run the controller in a port higher than 1000, creating a LoadBalancer service at port 80/443 and translating traffic into a higher targetPort to the controller:
E.g. setting such values via Helm chart:

controller:
  extraArgs:
    http-port: 10080
    https-port: 10443
  containerPort:
    http: 10080
    https: 10443
  service:
    externalTrafficPolicy: Local
    targetPorts:
      http: 10080
      https: 10443

This results in the source IP not being preserved, the ingress controller sees source IP as the service endpoint IP for the ingress controller, here is how the headers look like to the workload:

x-real-ip: INGRESS_ENDPOINT_IP_HERE
x-forwarded-for: INGRESS_ENDPOINT_IP_HERE
x-forwarded-host: demo.local
x-forwarded-port: "443"
x-forwarded-proto: "https"
x-forwarded-scheme: "https"

In AKS you don't have proxy protocol and you have to rely on this technique to pass the source IP to workloads.

[longwuyuan]

@pdefreitas unfortunately, it is true that AKS LB does not support proxy-protocol. Microsoft has mentioned in clearly and without any ambiguity, in their Azure/AKS docs.

[pdefreitas]

@longwuyuan yes I added that detail for awareness. In this scenario it fully limits which ports we are able to run HTTPS while preserving the source IP because X-Forwarded-Port is fixed to 443.

[anacelto]

We are facing this issue.

We have Azure Frontdoor -> Azure Load Balancer -> NGINX Ingress -> Keycloak.

Keycloak receives X-Forwarded-Port 80 when accessing via HTTPS.

Is there any standard/official workaround for this?

Currently, we configured Azure Frontdoor to set X-Forwarded-Port 443, and it works, but we are wondering if this is the correct solution.

[longwuyuan]

It will be good if I am corrected but I think all docs say that the X-Forwarded-For is received by the controller from the LB so the controller does not set that header by itself, or the other X-Forwarded headers like ip or proto or port.