Errors after upgrading from preview 7 to preview 8

[mrob1600]

I'm not able to see where the breaking change may be, but after upgrading to preview 8 all of our requests fail with the error below. I have included our config from appsettings. We aren't doing any config in code. Is it possible our route transforms aren't working like they did in preview 7?

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Microsoft.ReverseProxy.Service.Proxy.HttpProxy.ProxyAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, RequestProxyOptions requestOptions, HttpTransformer transformer)

Config:

"ReverseProxy": {
    "Routes": [
      {
        "RouteId": "PtmsDevReportingApiRoute",
        "ClusterId": "PtmsDevReportingCluster",
        "CorsPolicy": "AllowDevApiOrigins",
        "Match": {
          "Hosts": [ "ptms-dev.wakegov.com" ],
          "Path": "/api/reports/{**catch-all}"
        },
        "Transforms": [
          { "PathRemovePrefix": "/api" },
          {
            "RequestHeader": "Host",
            "Set": "services-ptms-dev.wakegov.com"
          }
        ]
      },
      {
        "RouteId": "PtmsDevApiRoute",
        "ClusterId": "PtmsDevCluster",
        "CorsPolicy": "AllowDevApiOrigins",
        "Match": {
          "Hosts": [ "ptms-dev.wakegov.com" ],
          "Path": "/api/{**catch-all}"
        },
        "Transforms": [
          { "PathRemovePrefix": "/api" },
          {
            "RequestHeader": "Host",
            "Set": "services-ptms-dev.wakegov.com"
          }
        ]
      },
      {
        "RouteId": "PtmsDevAuthRoute",
        "ClusterId": "PtmsDevCluster",
        "CorsPolicy": "AllowDevAuthOrigins",
        "Match": {
          "Hosts": [ "services-ptms-dev.wakegov.com" ],
          "Path": "/identity/{**catch-all}"
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsDevReportingServicesCatchAllRoute",
        "ClusterId": "PtmsDevReportingCluster",
        "CorsPolicy": "disable",
        "Match": {
          "Hosts": [ "services-ptms-dev.wakegov.com" ],
          "Path": "/reports/{**catch-all}"
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsDevServicesCatchAllRoute",
        "ClusterId": "PtmsDevCluster",
        "CorsPolicy": "disable",
        "Match": {
          "Hosts": [ "services-ptms-dev.wakegov.com" ],
          "Path": "{**catch-all}"
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsDevClientRoute",
        "ClusterId": "PtmsDevCluster",
        "CorsPolicy": "disable",
        "Match": {
          "Hosts": [ "ptms-dev.wakegov.com" ]
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsStagingApiRoute",
        "ClusterId": "PtmsStagingCluster",
        "Match": {
          "Hosts": [ "ptms-staging.wakegov.com" ],
          "Path": "/api/{**catch-all}"
        },
        "Transforms": [
          { "PathRemovePrefix": "/api" },
          {
            "RequestHeader": "Host",
            "Set": "services-ptms-staging.wakegov.com"
          }
        ]
      },
      {
        "RouteId": "PtmsStagingAuthRoute",
        "ClusterId": "PtmsStagingCluster",
        "CorsPolicy": "AllowStagingAuthOrigins",
        "Match": {
          "Hosts": [ "services-ptms-staging.wakegov.com" ],
          "Path": "/identity/{**catch-all}"
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsStagingServicesCatchAllRoute",
        "ClusterId": "PtmsStagingCluster",
        "Match": {
          "Hosts": [ "services-ptms-staging.wakegov.com" ],
          "Path": "{**catch-all}"
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      },
      {
        "RouteId": "PtmsStagingClientRoute",
        "ClusterId": "PtmsStagingCluster",
        "CorsPolicy": "disable",
        "Match": {
          "Hosts": [ "ptms-staging.wakegov.com" ]
        },
        "Transforms": [
          { "RequestHeaderOriginalHost": "true" }
        ]
      }
    ],
    "Clusters": {
      "PtmsDevCluster": {
        "Destinations": {
          "lrptmsdev1": {
            "Address": "https://lrptmsdev1/"
          },
          "lrptmsdev2": {
            "Address": "https://lrptmsdev2/"
          }
        }
      },
      "PtmsDevReportingCluster": {
        "Destinations": {
          "lrptmsrptdev1": {
            "Address": "https://lrptmsrptdev1/"
          }          
        }
      },
      "PtmsStagingCluster": {
        "Destinations": {
          "lrptmsstaging1": {
            "Address": "https://lrptmsstaging1/"
          },
          "lrptmsstaging2": {
            "Address": "https://lrptmsstaging2/"
          }
        }
      }
    }
  }

[Tratcher]

Interesting timing, we just released preview10 today 😁.
https://github.com/microsoft/reverse-proxy/releases/tag/v1.0.0-preview10

RemoteCertificateNameMismatch implies that the destination's certificate host name doesn't match the Host header on the request. Is that service expecting the public host or the internal one? Is see your setting "RequestHeaderOriginalHost": "true" to flow the public host on some routes, but using { "RequestHeader": "Host", "Set": "services-ptms-dev.wakegov.com" } on others? Which route is failing?

[mrob1600]

Yeah we've been stuck on preview 7 for awhile since I haven't had time to figure out the issue with upgrading to preview 8.

The routes that are failing are the ones with this transform { "RequestHeader": "Host", "Set": "services-ptms-dev.wakegov.com" }

Should this still work? We haven't changed the config or anything else in the environment except for upgrading to preview 8.

Thanks

[Tratcher]

The transforms infrastructure has been overhauled a few times so I may have broken you.

In Preview7 transforms were limited to one per header name and RequestHeaderOriginalHost avoided replacing another transform.
https://github.com/microsoft/reverse-proxy/blob/31ca104e18468e777c3787b48e097fb6e7cb28fb/src/ReverseProxy/Service/Config/TransformBuilder.cs#L537-L540

In preview8 I think the problem is here:
https://github.com/microsoft/reverse-proxy/blob/5b48e3727efd960d2c0719d67bd8df3c1cd19cf6/src/ReverseProxy/Service/Config/TransformBuilder.cs#L553-L556
If RequestHeaderOriginalHost is not set or is false, then a transform is added that removes the host header. The problem is that this is added and runs after your RequestHeader Set transform, so the Host gets set and then removed. You should be able to work around that by setting RequestHeaderOriginalHost to true.

If that works then I'll file an issue to address this.

[mrob1600]

That fixes it. Thanks!

This is the fix for now:

"Transforms": [
{ "PathRemovePrefix": "/api" },
{ "RequestHeaderOriginalHost": "true" },
{
"RequestHeader": "Host",
"Set": "services-ptms-dev.wakegov.com"
}
]

[Tratcher]

Filed #859