Requests-per-second metrics for each HTTP URL

[nfsp3k]

Hello,

Is it possible to distinguish requests-per-second based on HTTP URL using Linkerd proxy?
The workload coming to a frontend is 1000RPS, for example, and its composition is as follows:
GET /main -> 500 RPS
GET /cart -> 300 RPS
POST /cart -> 200 RPS

Is it possible to get this information using Linkerd proxy without modifying applications?
Assume that all the incoming traffic is not encrypted (using HTTP or gRPC, not HTTPS)

I guess it might be possible to get this information from the 'LIVE CALLS' information of Linkerd dashboard.
But, it seems some HTTP queries are missed here (Am I right?) and I want to get this information outside the cluster, which means it needs kind of REST APIs to get this.

Thank you in advance!!
It will be very helpful to my research!

[mateiidavid]

Hi @nfsp3k,

Have you tried looking at ServiceProfiles? It's a CRD that Linkerd uses to display per-route metrics, as well as to do retries and timeouts. It might be what you're looking for?

Here are some more details:

• ServiceProfiles docs
• Per-route metrics

[mateiidavid]

Hm, that's a bit odd. I see in tap there's a tls=not_provided_by_remote. Does the request originate from an ingress?

@grampelberg any ideas?

Edit:

The weird thing is that the 'linkerd top' command works for the frontend service:
image

It's been pointed out to me that top uses tap underneath, not ServiceProfiles, which would explain why it works for your service.

Can you do linkerd tap with -o json? Might give us a bit more insight into what's going on.

[nfsp3k]

Thanks again for your quick response, @Matei207!

Here is the link for linkerd tap -o json:
https://gist.github.com/nfsp3k/1373f973e85f6ec6946bec94ae05d0f6

I found it correctly works when I run the load generator INSIDE the cluster.

I originally ran a load generator named Locust outside of the cluster.
The requests were generated by the load gen and sent to the frontend directly.
(and linkerd routes was not working for this environment.)

However, it works when I run the load generator inside the cluster.
linkerd routes works fine as follows:

Service Profile is also working fine:

rt_route labels are also attached to the matched requests:

Here is the link for linkerd tap -o json when I run the load generator inside the cluster:
https://gist.github.com/nfsp3k/d21010f933b576f024a6d45f4da0b09b

I found the difference between the two environments is the existence of the header field l5d-dst-canonical.
For the first environment, the field l5d-dst-canonical did not exist in the requests that were generated by the load generator running outside the cluster. (the Linkerd proxy is also not attached to the load generator in this scenario.)

How can I fix it?
I'm going to use this feature for other applications too and some of them are not available to run load generator inside the cluster. Do I have to forcibly attach the header field to every generating request?
Thanks again!!

[ihcsim]

I originally ran a load generator named Locust outside of the cluster.

To get the route metrics using service profiles, your client needs to be injected with the Linkerd proxy. That's why it works when your load generator runs inside the cluster. The service profile doc has some info on how this works. If you want to keep the load generator outside of the cluster, you will have to add an ingress controller between the generator and your frontend service. See the Using Ingress on how to set it up.

[nfsp3k]

Thank you for your kind response, @ihcsim.
I will take a look at the docs.

And really thank you again for your quick responses, @Matei207!!

[nfsp3k]

May I ask you one more question, @ihcsim @Matei207?

I've still failed to collect per-route metrics even though I've been using ingress between load generator and my frontend service.
I guess the reason is that the Nginx Ingress Controller sets a bit weird value of the HTTP request header field 'Host' and 'l5d-dst-canonical'.
The detail is as follows.

I'm using a k8s Nginx Ingress Controller and applying Ingress resource to my k8s cluster.
The yaml file for Ingress is as follows:

The service is working well with the Ingress setting.
(The Nginx Ingress Controller Service is exposed by using NodePort.)

However, I still couldn't scrape per-route metrics in this scenario like this.

And I still couldn't find the rt_route label on the matched requests.

I think the reason is that the Nginx Ingress Controller modifies the value of the request header fields 'Host' and 'l5d-dst-canonical' in a bit weird way.
What I expect to the value of the fields 'Host' and 'l5d-dst-canonical' is 'frontend' and 'frontend.default.svc.cluster.local:80' respectively, however, the value set by Nginx Ingress Controller is "143.248.57.179:32721" for both of them.
You can see the result of linkerd tap with -o json in the following link: https://gist.github.com/nfsp3k/33ca2cefe9d47ad352436a37957617a7 (Line 38~)

When I removed the Linkerd proxy from the Nginx Pod, then the header field 'l5d-dst-canonical' disappeared from the HTTP requests and linkerd routes also doesn't work correctly.

My environment is as follows:

• Kubernetes Version: v1.17.9
• Cluster Environment: on-premise (1 master and 1 worker, Calico used for CNI)
• Host OS: Ubuntu 16.04
• Linkerd version: v2.8.1

Could you give me any comments about this?
Thanks in advance!!

[ihcsim]

@nfsp3k Your ingress issue sounds similar to kubernetes/ingress-nginx#5198. Try add the host field to the ingress resource.

[nfsp3k]

Thank you for the comment, @ihcsim.
I added the host field to the ingress resource and finally made it to function correctly.

I found its measurement speed is a bit slow.
When a load generator starts to send 500 RPS, for example, the time to fully detect the workload by linkerd route is several minutes, which seems too slow to me.

(I set the value of the field scrape_interval of the Linkerd-Prometheus Configmap to 1s.)

Can I accelerate the scraping speed?

[ihcsim]

How are you measuring the time linkerd route takes? If the time series data shows up in Prometheus, linkerd route should be able to pick them up right away.

[nfsp3k]

Thank you for your comment, @ihcsim.
What I missed when using linkerd route is the -t option. I used the default value of -t which is 1m and that's why the response speed of linkerd route seemed slow. Now, I set the value -t to 2 sec and it works perfectly except success rate.

I think its success rate seems odd.
In a case where a load generator sends 1000qps and the application can handle 500qps only, then what I expect to linkerd route is that it shows 500 qps of measured workload and 50% of success rate. However, the result showed 100% of the success rate with 500qps of the measured workload. I run linkerd routes svc/frontend -t 2s every second and the result is as follows:

• The blue line is the workload that the load generator sends.
• The green line shows the success rate of linkerd route.
• The red line is the workload that linkerd route measures.

Did I miss something or is it normal?

I really appreciate your kind comments, @ihcsim.
It has been very helpful to me.
Thank you again!!

[ihcsim]

Try to check the success rates from the client side, with linkerd routes svc/load-generator --to svc/frontend -t 2s.

and the application can handle 500qps only

Your frontend has to return a 5XX status code in order for Linkerd to mark that as a request failure.