Teleport 16.x dynamic app fails with 17.x backend

[programmerq]

Expected behavior:

When upgrading to Teleport 17.x, any 16.x app service agents that load dynamic app objects should continue working.

Current behavior:

When a 16.x agent is connected to a 17.x cluster, it fails to load dynamic apps. Instead, it prints a warning in a loop.

Re-init the cache on error: invalid origin value "aws-identity-center", must be one of [defaults config-file dynamic cloud kubernetes okta scim discovery-kubernetes entra-id]

Apps defined in the teleport.yaml load successfully. It's only dynamic apps that fail to load.

Note that there are no dynamic apps that have aws-identity-center as the origin. There are only app objects that have teleport.dev/origin: discovery-kubernetes. The new 17.x AWS Identity Center stuff was not configured on this 17.x cluster.

Bug Details

Hostname	Services	Agent Version
teleport-app-0	App,Discovery	v16.4.12
eps-proxy-5b8f485577-nkn7c	Proxy	v17.1.2
eps-auth-54467649b9-4q894	Auth,Kube	v17.1.2
eps-proxy-5b8f485577-fwzhk	Proxy	v17.1.2

Steps to recreate:

Create dynamic app resources that the 16.x app agent is supposed to load:

---
kind: app
metadata:
  description: Discovered application in Kubernetes cluster "mykube"
  labels:
    app.kubernetes.io/instance: echo
    app.kubernetes.io/name: echo
    teleport.dev/discovery-type: app
    teleport.dev/kubernetes-cluster: mykube
    teleport.dev/name: echo
    teleport.dev/origin: discovery-kubernetes
    teleport.dev/port: http
    teleport.internal/discovery-group-name: mykube
  name: echo
  revision: 8ed03e92-3142-9b9a-7338-b68217e1f7de
spec:
  insecure_skip_verify: false
  uri: http://echo.teleport.svc.cluster.local:8080
version: v3

Rendered `teleport.yaml` for the app agent:

app_service:
  apps:
    - name: dashboard
      rewrite:
        redirect:
          - localhost
      uri: http://traefik.traefik.svc.cluster.local:80/dashboard/
  debug_app: true
  enabled: true
  resources:
    - labels:
        teleport.dev/kubernetes-cluster: mykube
        teleport.dev/origin: discovery-kubernetes
auth_service:
  enabled: false
db_service:
  enabled: false
discovery_service:
  discovery_group: mykube
  enabled: true
  kubernetes:
    - labels:
        teleport.dev/name: '*'
        teleportcluster: eps.example.com
      namespaces:
        - '*'
      types:
        - app
jamf_service:
  enabled: false
kubernetes_service:
  enabled: false
proxy_service:
  enabled: false
ssh_service:
  enabled: false
teleport:
  join_params:
    method: kubernetes
    token_name: /etc/teleport-secrets/auth-token
  log:
    format:
      extra_fields:
        - timestamp
        - level
        - component
        - caller
      output: json
    output: stderr
    severity: DEBUG
  proxy_server: eps.example.com:443
version: v3

Logs:

{"caller":"cache/cache.go:1214","component":"app:service:1:1:cache","level":"debug","message":"Initiating new watch after waiting 5.462901512s.","timestamp":"2024-12-31T18:21:01Z"}
{"caller":"cache/cache.go:1199","component":"app:service:1:1:cache","level":"warning","message":"Re-init the cache on error: invalid origin value \"aws-identity-center\", must be one of [defaults config-file dynamic cloud kubernetes okta scim discovery-kubernetes entra-id]","timestamp":"2024-12-31T18:21:01Z"}
{"caller":"cache/cache.go:1203","component":"app:service:1:1:cache","level":"debug","message":"Reloading cache.","timestamp":"2024-12-31T18:21:01Z"}
{"caller":"cache/cache.go:1214","component":"app:service:1:1:cache","level":"debug","message":"Initiating new watch after waiting 26.304912851s.","timestamp":"2024-12-31T18:21:27Z"}
{"caller":"cache/cache.go:1199","component":"app:service:1:1:cache","level":"warning","message":"Re-init the cache on error: invalid origin value \"aws-identity-center\", must be one of [defaults config-file dynamic cloud kubernetes okta scim discovery-kubernetes entra-id]","timestamp":"2024-12-31T18:21:28Z"}
{"caller":"cache/cache.go:1203","component":"app:service:1:1:cache","level":"debug","message":"Reloading cache.","timestamp":"2024-12-31T18:21:28Z"}
{"caller":"cache/cache.go:1214","component":"app:service:1:1:cache","level":"debug","message":"Initiating new watch after waiting 24.0594765s.","timestamp":"2024-12-31T18:21:52Z"}
{"caller":"cache/cache.go:1199","component":"app:service:1:1:cache","level":"warning","message":"Re-init the cache on error: invalid origin value \"aws-identity-center\", must be one of [defaults config-file dynamic cloud kubernetes okta scim discovery-kubernetes entra-id]","timestamp":"2024-12-31T18:21:52Z"}

Once the agent is upgraded to Teleport 17.x the correct dynamic apps are loaded successfully.

[programmerq]

Workaround

Teleport 17.x ships a new role preset called role/aws-identity-center. This resource has teleport.dev/origin: aws-identity-center, which 16.x agents don't recognize. Edit that role and remove teleport.dev/origin: aws-identity-center entirely.

This is a valid workaround with the current (at time of writing) v16.4.12 and v17.1.2 versions for the agent and cluster.

Until a permanent fix is implemented, avoid enabling the AWS Identity Center Integration in your 17.1.2 cluster until all your agents have been upgraded from 16.x. That integration can create other resources that have the teleport.dev/origin: aws-identity-center label, which can lead to similar symptoms.

[zmb3]

This is fixed in 17.1.3 so long as the Identity Center integration is not enabled.