Linkerd-proxy rewrites host destionation port from 8084 to 80, making the request fail

[toonsevrin]

I am using Pomerium (an authentication proxy) to authenticate access to linkerd-web. Linkerd-web exposes HTTP on port 8084.

Without linkerd injection enabled in the pomerium namespace, things work fine. When I enable injection, linkerd-web is no longer reachable from my ingress.

After a 90+ message debugging session with the lovely @Matei207, we were able to track down the issue to this:

[  2684.437148497s] DEBUG outbound:accept{peer.addr=10.42.0.247:51686}:source{target.addr=10.43.220.50:8084}: linkerd2_app_outbound::endpoint: using host addr=linkerd-web.linkerd:80

Linkerd-proxy rewrites the host address to port 80, from 8084. This sounds like a bug.

Reproduction

To reproduce this setup a pod/service in a linkerd enabled namespace that both expose their http service on a port different from 80. Then simply send a request to the service. Enable debug and you will see something like:

DEBUG inbound:accept{peer.addr=10.42.0.244:55804}:source{target.addr=10.42.0.180:<<<PORT>>>}:http1{name=<<<host>>>:80 port=:<<<PORT>>>} keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=<<<host>>>:80}: linkerd2_service_profiles::client: name not in profile suffixes

[mateiidavid]

Hi, I was on slack with @toonsevrin and we couldn't reach a conclusion to the issue. It seems that the port is re-written to 80, the services had the correct ports listed in the spec and from what I have seen there isn't an issue with the headers. The request went from an ingress to the pomerium-proxy and finally to linkerd-web. I've attached some logs below.

• proxy logs with debug level for linkerd-web

4916.127031339s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}: linkerd2_proxy_transport::tls::accept: attempting TLS handshake
[  4916.128390218s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}: linkerd2_app_inbound::require_identity_for_ports: port=80 peer.id=Some("default.pomerium.serviceaccount.identity.linkerd.cluster.local") id_required=false
[  4916.129683482s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[  4916.130036958s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[  4916.130100774s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.42.0.227:80 self.port=4143
[  4916.130246121s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_cache: Dropping defunct service
[  4916.130311758s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}: linkerd2_cache: Caching new service
[  4916.135274896s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:poll_profile: linkerd2_service_profiles::client: profile received: DestinationProfile { routes: [], retry_budget: Some(RetryBudget { retry_ratio: 0.2, min_retries_per_second: 10, ttl: Some(Duration { seconds: 10, nanos: 0 }) }), dst_overrides: [] }
[  4916.139225873s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}: linkerd2_service_profiles::http::service: updating routes routes=0
[  4916.139568999s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:daemon:poll_profile: linkerd2_service_profiles::client: profile received: DestinationProfile { routes: [], retry_budget: Some(RetryBudget { retry_ratio: 0.2, min_retries_per_second: 10, ttl: Some(Duration { seconds: 10, nanos: 0 }) }), dst_overrides: [] }
[  4916.141194108s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}: linkerd2_service_profiles::http::service: updating routes routes=0
[  4916.141757222s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:target: linkerd2_cache: Caching new service
[  4916.142405568s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:target: linkerd2_proxy_http::client: method=GET uri=http://linkerd-web.linkerd/namespaces version=HTTP/1.1 headers={"host": "linkerd-web.linkerd", "user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0", "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "accept-language": "en-US,en;q=0.5", "accept-encoding": "gzip, deflate, br", "upgrade-insecure-requests": "1", "pragma": "no-cache", "cache-control": "no-cache", "te": "trailers", "cookie": "", "x-forwarded-for": "10.42.0.243,10.42.0.247", "x-forwarded-proto": "http", "x-request-id": "6796f1f9-47a2-4a15-8bc3-6a4cfdfbe940", "x-request-start": "t=1598012859.537", "content-length": "0", "x-envoy-external-address": "127.0.0.1", "x-envoy-expected-rq-timeout-ms": "30000", "l5d-dst-canonical": "linkerd-web.linkerd.svc.cluster.local:80"}
[  4916.142874927s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:target: linkerd2_proxy_transport::connect: Connecting peer.addr=127.0.0.1:80
[  4916.143523922s]  WARN inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[  4916.143869428s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_core::errors: Handling error with HTTP response status=502 Bad Gateway version=HTTP/2.0
[  4916.689265628s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[  4916.689613991s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[  4916.689674230s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.42.0.227:80 self.port=4143
[  4916.691727709s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:target: linkerd2_proxy_http::client: method=GET uri=http://linkerd-web.linkerd/favicon.ico version=HTTP/1.1 headers={"host": "linkerd-web.linkerd", "user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0", "accept": "image/webp,*/*", "accept-language": "en-US,en;q=0.5", "accept-encoding": "gzip, deflate, br", "pragma": "no-cache", "cache-control": "no-cache", "te": "trailers", "cookie": "", "x-forwarded-for": "10.42.0.243,10.42.0.247", "x-forwarded-proto": "http", "x-request-id": "0d956120-01bf-4a29-bfbd-71eeea5fd1f9", "x-request-start": "t=1598012860.139", "content-length": "0", "x-envoy-external-address": "127.0.0.1", "x-envoy-expected-rq-timeout-ms": "30000", "l5d-dst-canonical": "linkerd-web.linkerd.svc.cluster.local:80"}
[  4916.697144348s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}:http1{name=linkerd-web.linkerd.svc.cluster.local:80 port=80 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}:profile{addr=linkerd-web.linkerd.svc.cluster.local:80}:target: linkerd2_proxy_transport::connect: Connecting peer.addr=127.0.0.1:80
[  4916.697527681s]  WARN inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[  4916.697570381s] DEBUG inbound:accept{peer.addr=10.42.0.247:51316}:source{target.addr=10.42.0.227:80}: linkerd2_app_core::errors: Handling error with HTTP response status=502 Bad Gateway version=HTTP/2.0

• Tap

{
  "source": {
    "ip": "10.42.0.247",
    "port": 42378,
    "metadata": {
      "client_id": "default.pomerium.serviceaccount.identity.linkerd.cluster.local",
      "control_plane_ns": "linkerd",
      "deployment": "pomerium-proxy",
      "namespace": "pomerium",
      "pod": "pomerium-proxy-75c554bbc8-cz654",
      "pod_template_hash": "75c554bbc8",
      "serviceaccount": "default",
      "tls": "true"
    }
  },
  "destination": {
    "ip": "10.42.0.227",
    "port": 80,
    "metadata": {
      "control_plane_ns": "linkerd",
      "deployment": "linkerd-web",
      "namespace": "linkerd",
      "pod": "linkerd-web-b7c69668-sx2q4",
      "pod_template_hash": "b7c69668",
      "serviceaccount": "linkerd-web"
    }
  },
  "routeMeta": null,
  "proxyDirection": "INBOUND",
  "requestInitEvent": {
    "id": {
      "base": 1,
      "stream": 8
    },
    "method": "GET",
    "scheme": "",
    "authority": "linkerd-web.linkerd",
    "path": "/",
    "headers": [
      {
        "name": "host",
        "valueStr": "linkerd-web.linkerd"
      },
      {
        "name": "user-agent",
        "valueStr": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0"
      },
      {
        "name": "accept",
        "valueStr": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
      },
      {
        "name": "accept-language",
        "valueStr": "en-US,en;q=0.5"
      },
      {
        "name": "accept-encoding",
        "valueStr": "gzip, deflate, br"
      },
      {
        "name": "upgrade-insecure-requests",
        "valueStr": "1"
      },
      {
        "name": "pragma",
        "valueStr": "no-cache"
      },
      {
        "name": "cache-control",
        "valueStr": "no-cache"
      },
      {
        "name": "te",
        "valueStr": "trailers"
      },
      {
        "name": "cookie",
        "valueStr": ""
      },
      {
        "name": "x-forwarded-for",
        "valueStr": "10.42.0.243,10.42.0.247"
      },
      {
        "name": "x-forwarded-proto",
        "valueStr": "http"
      },
      {
        "name": "x-request-id",
        "valueStr": "332d0283-47ac-4071-a783-4ee374817bf1"
      },
      {
        "name": "x-request-start",
        "valueStr": "t=1598013520.325"
      },
      {
        "name": "content-length",
        "valueStr": "0"
      },
      {
        "name": "x-envoy-external-address",
        "valueStr": "127.0.0.1"
      },
      {
        "name": "x-envoy-expected-rq-timeout-ms",
        "valueStr": "30000"
      },
      {
        "name": "l5d-dst-canonical",
        "valueStr": "linkerd-web.linkerd.svc.cluster.local:80"
      }
    ]
  }
}

[grampelberg]

We're not rewriting the port, take a look at: "authority": "linkerd-web.linkerd"

This means that it is destined for linkerd-web.linkerd:80. If the port isn't included in authority, it is assumed to be the default (80).

You'll either want to correct the host or use l5d-dst-override.

[toonsevrin]

I see so if I'm correct the HOST header should be set to linkerd-web.linkerd:8084 on the HTTP request by the client? Seems like I should create a PR in Pomerium and not here 👍

[toonsevrin]

I still believe this should not try to route to port 80 should it? Like the logical behavior is that the header is preserved, but the port in the actual configuration is used.

Now this is extremely unlikely (as in usually impossible) but if you can somehow get in a HOST header with a spoofed port, one could potentially use it to route to ports that should not be accessible (say for example the linkerd-proxy control port!)

EDIT: This may be a very stupid threat model, my apologies.

[grampelberg]

Your ingress controller should also be filtering based off port. If your ingress definition is foobar.com and you have port 1234 listed in the definition, your ingress controller should return a 404 when the host header is foobar.com and not foobar.com:1234.

[toonsevrin]

Yep exactly! That's why I think this would be extremely unlikely to occur. That being said, bugs do happen and the ingress controller landscape is vast. If there is no trade-off, I'd always pick input that is defined in a kubernetes resource over a HTTP header (on the off-chance that an ingress controller does not check the HOST port)

[grampelberg]

We need to discover the k8s resource .... somehow. That's what the host header is for. What happens when someone has a service with the same name and two different ports ... how do we route then?

[toonsevrin]

Based on the TCP port 👍 but I can imagine that this could indeed cause unexpected behaviors in more complex situations. Thanks for explaining this, I'm very happy that we found the root of this issue and have already changed my service to port 80 (Seems like Pomerium is the culprit for stripping the host port).