http.client.duration vs redirections

[antonfirsov]

We are currently prototyping an implementation for http.client.duration in .NET (dotnet/runtime#84978 and dotnet/runtime#85447).

.NET's HttpClient deals with redirections automatically and transparently by default, meaning that the user will only see the last response in the redirect chain. Our problem is that considering that the tags scheme, host, protocol can change during redirects, it's not clear from the linked specification what should be logged and when.

Shall we log different http.client.duration entries for each internal request being made in the redirect chain? Or shall we log a single entry for the whole request chain? If it's the latter: what are the tag values to be logged, the ones from the first request or the ones from the final one?

[trask]

hi @antonfirsov!

ya, we probably need to specify this better.

to start with, can you clarify which modeling option you are using?

This span type represents an outbound HTTP request. There are two ways this can be achieved in an instrumentation:

1. Instrumentations SHOULD create an HTTP span for each attempt to send an HTTP request over the wire. In case the request is resent, the resend attempts MUST follow the HTTP resend spec. In this case, instrumentations SHOULD NOT (also) emit a logical encompassing HTTP client span.

2. If for some reason it is not possible to emit a span for each send attempt (because e.g. the instrumented library does not expose hooks that would allow this), instrumentations MAY create an HTTP span for the top-most operation of the HTTP client. In this case, the http.url MUST be the originally requested URL, before any HTTP-redirects that may happen when executing the request.

(https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md#http-client)

cc @lmolkova @mateuszrzeszutek

[antonfirsov]

which modeling option you are using?

@JamesNK is this something that has been decided? To me it looks like we could go with option 1, but I may be missing some background info.

[JamesNK]

@antonfirsov It's not been decided.

The linked content from @trask discusses tracing. HttpClient already has tracing. I'm pretty sure tracing happens in a HttpHandler and uses option 2. For consistency in approach and implementation, I think metrics should use option 2 as well.

Note that there is the option in the future to add different counters in SocketsHttpHandler that provide lower-level per-request counters if it's asked for.

@trask What are the implications of only having information on the topmost HTTP request for metrics?

[lmolkova]

I'm pretty sure tracing happens in a HttpHandler and uses option 2.

It happens in the handler right before SocketsHttpHandler and implements option 1 when it comes to retries. For redirects, it depends on user configuration.

Assuming we can and are going to introduce truly per-request metrics from the platform, providing clear http.client.duration that does not depend on user configuration would be great.

The implications of not doing it would be:

• it's impossible to tell if number of redirects is too big (or if there are any)
• it's impossible to tell which host/port was actually used and which one of them introduced higher error rate/latency

Option 2 is intended for instrumentation libraries (that can't overcome existing limitations) rather than native instrumentations which can do the right thing :)

[JamesNK]

You're right. I was only considering redirects. If there is a handler that retries (i.e. calls SendAsync multiple times) then those will be separate traces.

@antonfirsov My prototype has the counters in HttpMessageInvoker. Maybe they should be moved to DiagnosticsHandler?

[antonfirsov]

Yes it could be moved there, or to a separate handler on the same level. For me this is really a question whether we go with option 1 or 2. When looking at existing telemetry in System.Net.Http, counters and events go with option 2, but as @lmolkova mentioned DiagnosticscHandler does 1 for distributed tracing. I don't know what is the user expectation here, but I think it's easier to implement metrics correctly by choosing 1.

[lmolkova]

Option 1 provides better observability for HTTP calls. Assuming metrics measure latency/failure rate after all tries and redirects, they really measure how retry policy is configured rather than HTTP calls themselves.

This information is still useful, but it's a concern of different instrumentation layer (client library, ASP.NET Core HttpClientFactory, Polly, etc). For HTTP we need to cover the lower-level stuff if we can (and especially if it's easier) since there is nothing else that can cover it.