Add a way to customize X-Forwarded-For

[stac47]

Hello everyone,

I am trying to configure my ingress-nginx controller so that the applications
hosted in a Kubernetes cluster can see the real remote IP of the client.

My case is only based on X-Forwarded-For header and I cannot move the PROXY
protocol immediately.

Let's image the situation of application hosted in AWS that can be accessed in
different ways:

• a client can send a query to CloudFront which forwards to an Application Load
  Balancer
• a client can sent a query directly to an Application Load Balancer

This means the traversed trusted (inside AWS) proxies are either in the VPC
(second case) our outside of the VPC (first case).

More visually, let's take two clients:

       +----------+                     +----------+
       | Client 1 |                     | Client 2 |
       +----------+                     +----------+
          |                                  |
          |                                  |
+---------+---------------AWS----------------+----------------------+
|         |                                  |                      |
|         V                                  |                      |
|  +-------------+                           |                      |
|  | CloudFront  |                           |                      |
|  +-------------+                           |                      |
|         |                                  |                      |
|         |                                  |                      |
| +-------+--My VPC--------------------------+------------+         |
| |       |                                  |            |         |
| |       |                                  |            |         |
| |   +--------+                             |            |         |
| |   | API-GW |                             |            |         |
| |   +--------+                             |            |         |
| |       |                                  |            |         |
| |       +-------------------------------+  |            |         |
| |                                       |  |            |         |
| |                                       |  |            |         |
| |                                       V  V            |         |
| | +-----------EKS (k8s)----------+    +---------+       |         |
| | |                              |    | EC2 ELB |       |         |
| | |                              |    +---------+       |         |
| | |        +---------------+     |         |            |         |
| | |        | Nginx Ingress |<----+---------+            |         |
| | |        +---------------+     |                      |         |
| | |               |              |                      |         |
| | |               V              |                      |         |
| | |        +-------------+       |                      |         |
| | |        | Application |       |                      |         |
| | |        |   Pods      |       |                      |         |
| | |        +-------------+       |                      |         |
| | |                              |                      |         |
| | +------------------------------+                      |         |
| +-------------------------------------------------------+         |
+-------------------------------------------------------------------+

With the following configuration, as described in the documentation
(https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr):

use-forwarded-headers: "true"
proxy-real-ip-cidr: "${VPC_RANGE}"

The applications will receive the correct Client 2 IP address (because the
client is right at the boundary of the VPC). But the application will only see
the CloudFront machine IP when the requests are sent by Client 1.

One solution to cope with this issue could be to add the public IPs of
CloudFront that are published by AWS at
https://ip-ranges.amazonaws.com/ip-ranges.json. But that introduce some static
configuration and regular updates of the AWS IPs.

But a more dynamic solution could be to rely upon the CloudFront headers. For
Client 1, I think the best source of true should be in the added header
CloudFront-Viewer-Address.

So I was thinking to a solution in which I could create a
$custom_forwarded_for variable which would be used to set the correct value
to the X-Forwarded-For header to send to the application.

Something like this could do the trick:

map $http_CloudFront_Viewer_Address $custom_forwarded_for {
    "~*^(?<cf_viewer_address>.*):\d{1,5}$" $cf_viewer_address;

    default remote_addr;
}

The problem is that it cannot be used easily. The default template force this
line:

proxy_set_header            X-Forwarded-For        $remote_addr;

This line is as far as I know impossible to override. Using
more_set_input_headers will not help and reusing proxy_set_header would
only append the list.

So at that point I can see only solution:

• add a LUA plugin as discussed there
• use a custom template doc

I was wondering whether there was a more straightforward way to send the custom
X-Forwarded-For header.

I was thinking of an new configuration like custom-forwarded-for-variable. For
instance, we could have the following configuration in the ConfigMap:

http-snippet: |
  map $http_CloudFront_Viewer_Address $custom_forwarded_for {
      "~*^(?<cf_viewer_address>.*):\d{1,5}$" $cf_viewer_address;

      default remote_addr;
  }

custom_forwarded-for-variable: "custom_forwarded_for"

The official template could then be modified that way:

{{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
{{ $proxySetHeader }} X-Forwarded-For        $full_x_forwarded_for;
{{ else if $all.Cfg.CustomForwardedForVariable }}
{{ $proxySetHeader }} X-Forwarded-For ${{ $all.Cfg.CustomForwardedForVariable }};
{{ else }}
{{ $proxySetHeader }} X-Forwarded-For        $remote_addr;
{{ end }}

There are probably better solution that I would be happy to read.

I can see there are several issues related to mine (for instance), and think with this change, it would simply the management of other connectivities (like taking the CloudFlare's CF-Remote-IP header when needed).

Thanks in advance for your feedbacks or better solution to solve this usecase.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[strongjz]

@stac47 There is another load balancer when you deploy it into AWS that would need the settings in the ELB like the other one. Ingress deploys a service type of LoadBalancer and the cloud controller deploys one for Ingress to use.

[jetersen]

@strongjz I have a very similar case. I want to support both NLB using (proxy protocol) and ALB using client port preservation, but use-forwarded-headers behavior is odd ie. it removes the client port. We often need this for government/police requests. Potentially in the future I will also need to support CloudFront 😅

So far, I have had to use two separate nginx ingress for ALB and NLB (it depends on use case)

For ALB

    proxySetHeaders:
      X-Forwarded-Port: '$remote_port'
      X-Forwarded-For: '$remote_addr:$remote_port'

For NLB

    proxySetHeaders:
      X-Forwarded-Port: '$proxy_protocol_port'
      X-Forwarded-For: '$proxy_protocol_addr:$proxy_protocol_port'

For .NET we need the addr:port because how .NET parses forwarded for header.

With proxy protocol the x-forwarded-for gets an ip at least without the use-forwarded-headers=false and use-proxy-protocol=true

with ALB and client port preservation the X-Original-Forwarded-For is fine and has a port but than X-Forwarded-For seems to remove the port with use-forwarded-headers=true
So that's why I added the proxySetHeaders

So yes it seems very much like I want some sort of customization that allows me for one to have conditional so I could use the same load balancer with minimal configuration change.

Example of the headers:

        "X-Forwarded-For": [
            "10.10.20.139",
            "10.10.20.139:22737"
        ],
        "X-Forwarded-Host": [
            "host.kube.my"
        ],
        "X-Forwarded-Port": [
            "443",
            "22737"
        ],
        "X-Forwarded-Proto": [
            "https"
        ],
        "X-Forwarded-Scheme": [
            "https"
        ],

        "X-Forwarded-For": [
            "99.99.99.123",
            "99.99.99.123:24800"
        ],
        "X-Forwarded-Host": [
            "host.kube.my"
        ],
        "X-Forwarded-Port": [
            "443",
            "24800"
        ],
        "X-Forwarded-Proto": [
            "https"
        ],
        "X-Forwarded-Scheme": [
            "https"
        ],

[stac47]

@strongjz Hello, I updated the schema to be closer to my infrastructure.

There is another load balancer when you deploy it into AWS that would need the settings in the ELB like the other one. Ingress deploys a service type of LoadBalancer and the cloud controller deploys one for Ingress to use.

Not sure to understand. Do you mean my proposal would not work?

[zlodes]

We have slightly the same issue, but with CloudFlare, not CloudFront.

Temporary solution is splitting Ingresses and add annotation to the second one (for clients with CF):

nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-For $http_cf_connecting_ip;

Similar issues:

1. Cloudflare + ELB extracting the client IP in nginx ingress and whiltelist based on the client IP #7761
2. Cloudflare & Proxy Protocol #4731
3. Whitelist Annotation with CDN (Cloudflare) infront not working #3738