Some gateways are not alive with inlets on target cluster

[devsentient]

I'm trying to mirror services from a local cluster that's exposed via the Inlets operator to a GKE cluster. After following the tutorial here:
https://linkerd.io/2/tasks/multicluster/

I'm getting the following validation error:

× all target cluster gateways are alive
    Some gateways are not alive:
    * cluster: [xxxxx], gateway: [linkerd-multicluster/linkerd-gateway]
    see https://linkerd.io/checks/#l5d-multicluster-target-gateways-alive for hints

Does this have something to do with the fact that Inlets only exposes a single port?

On the GKE cluster's linkerd-service-mirror pod I'm getting lots of the following error:

WARN outbound:accept{peer.addr=xx.xx.xx.xx:50138}:source{target.addr=xx.xx.xx.xx:4181}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast

Any thoughts?

[ihcsim]

So your target cluster opens an inlet tunnel, instead of launching an external load balancer? (I am assuming this is a private GKE cluster?) Does the external IP of the target linkerd-gateway service matches what the service-mirror thinks it is? Runnnig kubectl -n linkerd-multicluster get svc on the target cluster should show whether the gateway has an external IP. On the source cluster, the service-mirror container usually have some useful logs on where it thinks the target gateway is.

[devsentient]

Cluster A is on GKE, cluster B is a physical server behind a firewall. Cluster B only has SSH open + 2 other open ports. Trying to figure out how to mirror services from cluster B to cluster A. Inlets doesn't seem to cut it.

[ihcsim]

Did linkerd-gateway get an external IP? If yes, is that external IP accessible from cluster A?

linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast

This is likely irrelevant.

[devsentient]

It does have an external IP and it's accessible, but only has 3 ports open. Two of the three ports are taken up by SSH and the kubernetes API endpoint, so only one port available for Linkerd.

[ihcsim]

For multicluster to work, you will need to open port 4143 and 4181, as defined in the linkerd-gateway service.

As for inlets, keep in mind that by default, the exit server listens on port 80, but linkerd check --multicluster uses the probe port at 4181 to determine if the remote gateway is alive. Your GKE cluster should have an Endpoint resource with endpoint info of your remote gateway. In the recent edge releases, we introduced a new Link CRD to make configuring the gateway address, port and probe spec easier.

PS I changed the name of the probe port to http per the inlets doc, but that didn't change the port that the exit server listens on.