Tracing add-on: Failed to proxy request: Connection refused (os error 111)

[byakku]

Bug Report

What is the issue?

Proxy containers have connection refused errors, can't get tracing with Jaeger working.

How can it be reproduced?

1. Install linkerd (cli, step + CertManager)
2. Enable tracing
3. Patch deployment

kubectl -n default patch deployment some-app -p '
spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/trace-collector: linkerd-collector.linkerd.svc.cluster.local:55678
        config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector
'

Logs, error output, etc

kubectl logs -f some-app -c linkerd-proxy

[     8.863549040s]  WARN inbound:accept{peer.addr=10.10.71.30:48544}:source{target.addr=10.10.150.75:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[    38.860966335s]  WARN inbound:accept{peer.addr=10.10.71.30:48544}:source{target.addr=10.10.150.75:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[    68.868491316s]  WARN inbound:accept{peer.addr=10.10.71.30:48544}:source{target.addr=10.10.150.75:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 66630.370550855s]  WARN inbound:accept{peer.addr=10.10.135.15:55624}:source{target.addr=10.10.150.75:8080}: linkerd2_app_core::errors: Failed to proxy request: connection error: Connection reset by peer (os error 104)
[ 66638.861042031s]  WARN inbound:accept{peer.addr=10.10.71.30:48544}:source{target.addr=10.10.150.75:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[ 66668.860884682s]  WARN inbound:accept{peer.addr=10.10.71.30:48544}:source{target.addr=10.10.150.75:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

target.addr = app-container of patched deployment

linkerd check output

linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
× can query the control plane API
    Post "http://localhost:45123/api/v1/Version": dial tcp: lookup localhost on 1.1.1.1:53: no such host
    see https://linkerd.io/checks/#l5d-existence-api for hints

Status check results are ×

But when I port-forward API:

kubectl -n linkerd port-forward \
    $(kubectl -n linkerd get po \
        --selector=linkerd.io/control-plane-component=controller \
        -o jsonpath='{.items[*].metadata.name}') \
9995:9995
Forwarding from 127.0.0.1:9995 -> 9995
Forwarding from [::1]:9995 -> 9995

I can access metrics:

curl -s 127.0.0.1:9995/metrics | head -n 4
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.5951e-05
go_gc_duration_seconds{quantile="0.25"} 5.9676e-05

When using cli with --api-addr parameter:

linkerd --api-addr 127.0.0.1:9995 -n default check --proxy
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
× can query the control plane API
    HTTP error, status Code [404] (unexpected API response: 404 page not found
)
    see https://linkerd.io/checks/#l5d-existence-api for hints

Status check results are ×

Environment

• Kubernetes Version: 1.14.7
• Cluster Environment: EKS
• Host OS: Amazon Linux
• Linkerd version: Client version: stable-2.8.1
• Network layer: AWS CNI

Additional context

Linkerd deployment

kubectl -n linkerd get pods
NAME                                      READY   STATUS    RESTARTS   AGE
linkerd-collector-b6974f8f7-zxtjz         2/2     Running   0          19h
linkerd-controller-7679568559-58kw7       2/2     Running   0          20h
linkerd-destination-5645c59bb-8brv4       2/2     Running   0          20h
linkerd-grafana-7747dcc6ff-47rs6          2/2     Running   0          20h
linkerd-identity-558dfc5957-7nqxp         2/2     Running   0          20h
linkerd-jaeger-58d77f9fb5-rkf62           2/2     Running   0          19h
linkerd-prometheus-7fc659b5bf-99ppl       2/2     Running   0          20h
linkerd-proxy-injector-74bf4fc4bb-kzgdn   2/2     Running   0          20h
linkerd-sp-validator-5ffbb6dccc-prw2p     2/2     Running   0          20h
linkerd-tap-5b9f58947b-ch22s              2/2     Running   0          20h
linkerd-web-6cd854c4b7-rkrhm              2/2     Running   0          19h

Short version of app deployment configuration:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "44"
  generation: 63
  labels:
    app: my-app
    chart: app-0.1.0
    heritage: Tiller
    release: my-app
  name: my-app
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
      release: my-app
  template:
    metadata:
      annotations:
        config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector
        config.linkerd.io/admin-port: "8082"
        config.linkerd.io/control-port: "8081"
        config.linkerd.io/trace-collector: linkerd-collector.linkerd.svc.cluster.local:55678
        linkerd.io/inject: enabled
      creationTimestamp: null
      labels:
        app: my-app
        release: my-app
        version: redacted
    spec:
      containers:
      - args:
        - -jar
        - /app.jar
        command:
        - java
        env:
        [some envs]
        - name: OC_AGENT_HOST
          value: linkerd-collector.linkerd.svc.cluster.local:55678
        image: redacted
        imagePullPolicy: IfNotPresent
        name: app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 1099
          name: jmx
          protocol: TCP
        - containerPort: 5005
          name: debug
          protocol: TCP
        - containerPort: 9000
          name: metrics
          protocol: TCP
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler

What am I missing?

[byakku]

Okay, I did clean install

linkerd install | kubectl apply -f -

Injected tracing config

Patched deployment

kubectl -n default patch deployment some-service -p '
spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/trace-collector: linkerd-collector.linkerd.svc.cluster.local:55678
        config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector
'

Now when I'm checking logs in proxy container in injected pod, it tells me about wrong certificate

[   116.255227540s]  WARN outbound:accept{peer.addr=10.10.124.1:38040}:source{target.addr=172.20.214.212:80}:logical{addr=some-service.default.svc.cluster.local:80}:profile:balance{addr=some-service.default.svc.cluster.local:80}:endpoint{peer.addr=10.10.146.168:8080}: rustls::session: Sending fatal alert BadCertificate    

linkerd check

linkerd --api-addr 127.0.0.1:8085 check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

linkerd-tracing
---------------
√ collector service account exists
√ jaeger service account exists
√ collector config map exists
√ collector pod is running
√ jaeger pod is running

Status check results are √

[byakku]

Another finding - I've got ancient cert-manager version deployed. After upgrading, reinstalling linkerd with --identity-external-issuer fixed issues with certs.

Timeouts

[    20.27736197s]  WARN inbound:accept{peer.addr=10.10.71.30:59556}:source{target.addr=10.10.137.180:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[    50.26497637s]  WARN inbound:accept{peer.addr=10.10.71.30:59556}:source{target.addr=10.10.137.180:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[    55.760113753s]  WARN inbound:accept{peer.addr=10.10.83.47:53966}:source{target.addr=10.10.137.180:8080}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)
[    80.26520083s]  WARN inbound:accept{peer.addr=10.10.71.30:59556}:source{target.addr=10.10.137.180:9000}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)

Only appear after launch, later on I have no errors with certs/connections.

The only issue I have right now is tracing that does not really work.

Annotations:

config.linkerd.io/trace-collector: linkerd-collector.linkerd.svc.cluster.local:55678
config.alpha.linkerd.io/trace-collector-service-account: linkerd-collector

With env:

OC_AGENT_HOST=linkerd-collector.linkerd:55678

on multiple deployments, unfortunately nothing appears in Jaeger, the only one service I can select is jaeger-query

What could be misconfigured?

[ihcsim]

The OC_AGENT_HOST is only relevant to the sample emojivoto application. If you haven't already, try getting tracing to work with that application first. See documentation here. Otherwise, with so many components involved, it's hard to pinpoint where the error is.

[byakku]

I made tracing work with emojivoto app, so I assume the components are working.

[byakku]

Nevermind - tracing has been disabled on our services. 🤦‍♂️

Issue can be closed.