Traefik v2 allowing public access to port 80 and 443 but not 50051 (grpc)

[belugame]

traefik version
Version:      2.11.10
Codename:     mimolette
Go version:   go1.23.1
Built:        2024-09-19T10:11:01Z
OS/Arch:      linux/amd64
k3s version v1.31.4+k3s1 (a562d090)
go version go1.22.9

I struggle hard getting my k3s/traefik v2 setup to work for my seemingly very simple usecase. This is on a Debian 12 bare metal server.

What works:

• port 80 and 443 answer on desired domain and the requests are correctly routed to my django pod
• port 50051 for a grpc server answers on pod and service level
• the traefik pod shows 50051 as open
• traffic on 80/443 vs on 50051 are dealt with two different ingress routes and host matching (2 subdomains)

What doesn't work:
Port 50051 for a grpc server answers on pod and service level but cannot be reached from the outside world. I tried with nc and grpcurl which work from inside but they cannot connect from outside.

In general I struggle hard to have a consistent grip over the k3s config. Sometimes it seems like it ignores or forgets my config changes. E.g. trying to have debug log by adding these lines to valuesContent of traefik has no effect (checking description of the pod).
Even when I force a rollout with: k3s kubectl rollout restart deployment traefik -n kube-system

log:
level: "DEBUG"
general:
    level: "DEBUG"
additionalArguments:
- "--log.level=DEBUG"

When I kill the pod and it is recreated I see only: time="2025-01-23T13:20:23Z" level=info msg="Configuration loaded from flags."

My endpoints:

k3s kubectl get endpoints  -n kube-system
NAME             ENDPOINTS                                       AGE
kube-dns         10.42.0.3:53,10.42.0.3:53,10.42.0.3:9153        26h
metrics-server   10.42.0.2:10250                                 26h
traefik          10.42.0.33:80,10.42.0.33:50051,10.42.0.33:443   14h

My traefik config:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
    image:
    repository: "rancher/mirrored-library-traefik"
    tag: "2.11.10"
    deployment:
    replicas: 1
    ports:
      web:
        port: 80
        expose:
          default: true
      websecure:
        port: 443
        expose:
          default: true
      grpc:
        port: 50051
        expose:
          default: true

My ingress routes:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: http-ingressroute
  namespace: myapp-staging
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`staging.myapp.domain.com`)
      kind: Rule
      services:
        - name: django
          port: {{ .Values.django.ports.http }}  # = 8000
          scheme: http
          namespace: myapp-staging
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: grpc-ingressroute
  namespace: myapp-staging
spec:
  entryPoints:
    - grpc
  routes:
    - match: Host(`grpc.myapp.domain.com`)
      kind: Rule
      services:
        - name: grpc
          port: {{ .Values.django.ports.grpc }}  # = 50051
          scheme: h2c
          namespace: myapp-staging

My grpc service:

---
apiVersion: v1
kind: Service
metadata:
  name: grpc
  labels:
    app: django
spec:
  selector:
    app: django
  ports:
    - port: {{ .Values.django.ports.grpc }}
      targetPort: {{ .Values.django.ports.grpc }}

My grpc container:

  - name: grpc-server
        image: {{ .Values.django.image.main }}
        command: ["python", "-u", "grpc_app/server.py"]
        workingDir: /app/myapp.git
        ports:
        - containerPort: {{ .Values.django.ports.grpc }}
        volumeMounts:
        - name: app-code
          mountPath: /app
        envFrom:
          - configMapRef:
              name: django-env-variables
        env:
          - name: CELERY_BROKER_URL
            value: "redis://redis:{{ .Values.redis.port }}/0"
          - name: CELERY_RESULT_BACKEND
            value: "redis://redis:{{ .Values.redis.port }}/1"
          - name: NATS_URL
            value: "nats://nats:{{ .Values.nats.port }}"
      volumes:
      - name: app-code
        emptyDir: {}

My traefik pod:

k3s kubectl describe pod traefik -n kube-system
Name:             traefik-6c4d764f45-zw2tf
Namespace:        kube-system
Priority:         0
Service Account:  traefik
Node:             myserver/10.5.50.10
Start Time:       Thu, 23 Jan 2025 13:27:33 +0100
Labels:           app.kubernetes.io/instance=traefik-kube-system
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=traefik
                  helm.sh/chart=traefik-27.0.201_up27.0.2
                  pod-template-hash=6c4d764f45
Annotations:      prometheus.io/path: /metrics
                  prometheus.io/port: 9100
                  prometheus.io/scrape: true
Status:           Running
IP:               10.42.0.33
IPs:
  IP:           10.42.0.33
Controlled By:  ReplicaSet/traefik-6c4d764f45
Containers:
  traefik:
    Container ID:  containerd://408ecb22b243e52e36c3aa2b2c7c3edbe27c1c20bf4ac22f65e8bda771d691e8
    Image:         rancher/mirrored-library-traefik:2.11.10
    Image ID:      docker.io/rancher/mirrored-library-traefik@sha256:b19934b3f8e03e213e258e69d57376fd0c4441ea04a518313cab2835fe9b92b3
    Ports:         50051/TCP, 9100/TCP, 9000/TCP, 80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.grpc.address=:50051/tcp
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:80/tcp
      --entrypoints.websecure.address=:443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --entrypoints.websecure.http.tls=true
    State:          Running
      Started:      Thu, 23 Jan 2025 13:27:33 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:
      POD_NAME:       traefik-6c4d764f45-zw2tf (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mvk6 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-5mvk6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  18m   default-scheduler  Successfully assigned kube-system/traefik-6c4d764f45-zw2tf to myserver
  Normal  Pulled     18m   kubelet            Container image "rancher/mirrored-library-traefik:2.11.10" already present on machine
  Normal  Created    18m   kubelet            Created container traefik
  Normal  Started    18m   kubelet            Started container traefik

[brandond]

Sometimes it seems like it ignores or forgets my config changes. E.g. trying to have debug log by adding these lines to valuesContent of traefik has no effect (checking description of the pod).

My traefik config:

apiVersion: helm.cattle.io/v1
kind: HelmChart

Don't edit the HelmChart. Packaged components are reverted to their default content every time k3s is restarted.

Use a HelmChartConfig to add values as shown in the documentation. https://docs.k3s.io/helm#customizing-packaged-components-with-helmchartconfig

[belugame]

On the host I cannot see anything that would block it. For confirmation if there is something between the host and the world I need to wait for a colleague to confirm.
But just for trying out I wanted to disable the web and websecure endpoints and put grpc on port 80, but I again can't get the traefik to update. Here is what I do via ansible:

- name: Manage Traefik in K3S
  hosts: k3s-server
  gather_facts: no
  become: yes
  vars:
    venv_folder: "/tmp/ansible-env"
  tasks:
    - name: Set ansible_python_interpreter to the virtual environment's python
      ansible.builtin.set_fact:
        ansible_python_interpreter: /tmp/ansible-env/bin/python

    - name: Write Traefik configuration to HelmChartConfig file
      ansible.builtin.copy:
        dest: /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
        content: |
          apiVersion: helm.cattle.io/v1
          kind: HelmChartConfig
          metadata:
            name: traefik
            namespace: kube-system
          spec:
            valuesContent: |-
              image:
                repository: "rancher/mirrored-library-traefik"
                tag: "2.11.10"
              deployment:
                replicas: 1
              ports:
                grpc:
                  port: 80
                  expose:
                    default: true
              log:
                level: "DEBUG"

And I can see that the file changes on the host (and sometimes it does update). But now there is no update in the pods even if I rollout traefik newly or delete the pod or run k3s kubectl apply -f traefik-config.yaml (returns helmchartconfig.helm.cattle.io/traefik configured). Could you point out what I'm still missing in terms of updating config?

[brandond]

I don't see that you're disabling web in the values. Don't you now have both web and grpc on the same port? I don't think that'll work.

[belugame]

I mean I removed the 80 and 443 ports and have grpc on 80 now while also removing the django (http) service and ingress. But my traefik still shows the grpc:50051 port despite the update :/

[brandond]

You need to disable the web/websecure ones as they are still in the default values. You can't just not specify them in the custom values. You have to remember that helm merges everything.

[belugame]

Hey Brandon, I learnt that 50051 is blocked for the server from the outside (which I did not know :() and I need to use 80/443 anyway for client restrictions. So you were right. I now don't have need for a custom traefik config and I got it running on 80 for time being with a simplified ingress:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: http-ingressroute
  namespace: energycloud-staging
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`www.somedomain.com`)
      kind: Rule
      services:
        - name: django
          port: {{ .Values.django.ports.http }}
          scheme: http
    - match: Host(`grpc.somedomain.com`)
      kind: Rule
      services:
        - name: grpc
          port: {{ .Values.django.ports.grpc }}
          scheme: h2c

Still hope that I can solve the inconsistent config-reloading but can close this issue.
Thank you 🙏

[belugame]

Problem was not with my config but that the server was configured from the outside to only receive traffic on 80 and 443.