OAuth flow failed while using SecurityPolicy with EnvoyGateway

[HannaManista]

Title: OAuth flow failed while using multiple authentication sessions with EnvoyGateway

Description:
I set up 2 separate SecurityPolicies (using api: gateway.envoyproxy.io/v1alpha1) for each of my HTPRoutes (from gateway.networking.k8s.io/v1). Both of the SecurityPolicies point to the same Azure application. Their configuration is as follows:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: "prometheus-oidc" 
  namespace: monitoring
spec:
  oidc:
    clientID: {{ .Values.authentication.clientID }} # application ID of the EntraID OIDC application 
    clientSecret:
      group: ""
      kind: Secret
      name: {{ .Values.authentication.clientSecretName }} 
    logoutPath: /logout
    provider:
      issuer: https://login.microsoftonline.com/<tetnant-ID>/v2.0
    redirectURL: https://{{ .Values.prometheus.dnsZoneName }}/oauth2/callback 
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: {{ .Values.prometheus.httpRouteName }}

Firstly I set only one of the SecurityPolicies to authenticate the traffic to the HTTPRoute. It worked - routed to the login page and after logging in to the service page.
I wanted to enable the second SecurityPolicy for another HTTPRoute (another service). After its successful creation I called the service's url, was once again redirected to the authentication page, but after logging in I get the error page: "OAuth flow failed."
In the Envoy gateway's pod the log looks as following:

the only information here is that it is a 401 error code, while with the same credentials I am authenticated by the same Azure application to another service (using different HTTPRoute).

The same behavior was observed when attaching 2 HTTPRoutes to one SecurityPolicy.
Also what is the way to track the authorized OIDC sessions in SecurityPolicy provided by EnvoyGateway? (Analogically to oauth2 which has redis.)

Config:
Envoy is installed using helm chart gateway-helm from docker.io/envoyproxy , chart version 1.2.1.
EnvoyProxy customization is only annotations:

apiVersion: gateway.envoyproxy.io/v1alpha1

kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: {{ .Values.gateway.namespace }}
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          service.beta.kubernetes.io/azure-load-balancer-internal-subnet: {{ .Values.gateway.loadbalancer.subnet }}

[arkodg]

cc @zhaohuabing

[zhaohuabing]

@HannaManista Could you please paste your Gateway, SPs and HTTPRoutes resoures?

You can substitue the sensitive info with some placeholders, but please keep the original configurations as much as possible.

[zhaohuabing]

Both of the SecurityPolicies point to the same Azure application

I think you can't do that because the redirect urls for different SPs should not be the same.

https://gateway.envoyproxy.io/docs/tasks/security/oidc/#register-an-oidc-application

[HannaManista]

Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-gateway
  namespace: ingress
  annotations:
    cert-manager.io/cluster-issuer: {{ .Values.gateway.certificate.clusterIssuer }}
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: {{ .Values.gateway.loadbalancer.subnet }}
  labels:
    app.kubernetes.io/component: proxy
    app.kubernetes.io/managed-by: "envoy-gateway-controller" 
    app.kubernetes.io/name: "envoy" 
    gateway.envoyproxy.io/owning-gateway-name: {{ .Values.gateway.name }}
    gateway.envoyproxy.io/owning-gateway-namespace: {{ .Values.gateway.namespace }}
spec:
  gatewayClassName: "{{ .Values.gateway.namespace }}-gateway-controller"
  addresses:
    - type: IPAddress
      value: {{ .Values.gateway.privateIP }}
  listeners:
    - name: http
      port: {{ .Values.gateway.httpPort }}
      protocol: HTTP
      hostname: "*.{{ .Values.gateway.dnsName }}"
      allowedRoutes: 
        namespaces:
          from: All
    - name: https-1
      port: {{ .Values.gateway.httpsPort }}
      protocol: HTTPS
      hostname: "*.{{ .Values.gateway.dnsName }}"
      allowedRoutes: 
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
        - group: ''
          kind: Secret
          name: "{{ .Values.gateway.namespace }}-wildcard-tls-cert" 
          namespace: {{ .Values.gateway.namespace }}
    - name: https
      port: {{ .Values.gateway.httpsPort }}
      protocol: HTTPS
      hostname: "{{ .Values.gateway.dnsName }}"
      allowedRoutes: 
        namespaces: 
          from: All
      tls:
        mode: Terminate
        certificateRefs:
        - group: ''
          kind: Secret
          name: "{{ .Values.gateway.namespace }}-wildcard-tls-cert" 
          namespace: {{ .Values.gateway.namespace }}

HTTPRoute for app1:

 apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus-stack
spec:
  hostnames:
  - {{ required "dnsZoneName is required" .Values.prometheus.dnsZoneName }}
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: envoy-gateway
    namespace: ingress 
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: {{ .Values.prometheus.serviceName }}
      namespace: monitoring 
      port: 9090
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

HTTPRoute for appp2:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: alertmanager
  namespace: monitoring
  labels:
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/part-of: kube-prometheus-stack
spec:
  hostnames:
  - {{ required "dnsZoneName is required" .Values.alertmanager.dnsZoneName }}
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: envoy-gateway
    namespace: ingress 
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: {{ .Values.alertmanager.serviceName }}
      namespace: monitoring 
      port: 9093
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

SecurityPolicy for app1:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: "prometheus-oidc" 
  namespace: monitoring
spec:
  oidc:
    clientID: {{ .Values.authentication.clientID }} # application ID of the EntraID OIDC application 
    clientSecret:
      group: ""
      kind: Secret
      name: {{ .Values.authentication.clientSecretName }} #oauth2-proxy-client
    logoutPath: /logout
    provider:
      issuer: https://login.microsoftonline.com/<tenant-ID>/v2.0
    redirectURL: https://{{ .Values.prometheus.dnsZoneName }}/oauth2/callback #redirectURI from the EntraID OIDC application 
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: prometheus

SecurityPolicy for app2:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: "alertmanager-oidc" 
  namespace: monitoring
spec:
  oidc:
    clientID: {{ .Values.authentication.clientID }} # application ID of the EntraID OIDC application 
    clientSecret:
      group: ""
      kind: Secret
      name: {{ .Values.authentication.clientSecretName }} #oauth2-proxy-client
    logoutPath: /logout
    provider:
      issuer: https://login.microsoftonline.com/<tenant-ID>/v2.0
    redirectURL: https://{{ .Values.alertmanager.dnsZoneName }}/oauth2/callback #redirectURI from the EntraID OIDC application 
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: alertmanager

Concerning the redirect URLs for both SPs, in the Azure Application it is possible to set multiple (different) redirect URIs:

It doesn't say that the authentication application and SP should be 1-1, as I understand.

[zhaohuabing]

@HannaManista I'v tested a similar setup: two ScecurityPolicies configured with the same application id targeting on two HTTPRoutes with different hosts, and it worked as expected.

If you could enable the debug log on the Envoy and share the log, I'd be happy to help investigate and debug this issue further.

Debug log can be enabled through EnvoyProxy:

kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
    logging:
    level:
      default: debug

[HannaManista]

Hi @zhaohuabing , I enabled the logging level to debug on EnvoyProxy. After calling the service in envoy-gateway's pod logs there is:

and here another request. For some reason it looks different:

based on this error:
"error":"invalid_request","error_description":"AADSTS900144: The request body must contain the following parameter: 'client_id'
it looks like the client_ID of the Authenticating Azure application is not passed, but the problem is that both of the applications have a shared values and get the client_ID to this application from the same value:
value file:

the SecurityPolicy for both apps:

[HannaManista]

this is the log from the successfull authentication of the second service:

[HannaManista]

One more observation regarding those two SecurityPolicies: this time I set up the SecurityPolicy for alertmanager firstly and afterward SecurityPolicy for prometheus was created. Result is that SecurityPolicy for alertmanager works correctly whereas the SecurityPolicy for (secondly created) prometheus fails on authentication. In my earlier investigation I created the prometheus SecurityPolicy firstly and it worked, while secondly (at that time) created alertmanager SecurityPolicy failed. So the order of creation here might be crucial, not really the service itself.