thoughts on trace inferred metrics?

[naseemkullah]

seems like these are becoming a thing, suppose sampling rate is 1%, total request count is inferred as trace-count * 100. How, if at all, would this fit into linkerd?

[ihcsim]

Can you give me an idea that kind of use cases and problems that this will solve? References to some links and docs will be helpful. (My search doesn't yield anything useful.) Does the inference provide some form of projection of when certain thresholds, SLOs etc. will be violated? I assume this will require some code change on the user application?

[naseemkullah]

BTW, I just heard about this concept today after attending an office hours with an engineer at Honeycomb.

I think this is just a drop-in replacement to metrics collection, since for debugging traces provide more context and detail, but since they are more expensive to collect and store you still need metrics for things like total counts and such. So basically instead of collecting metrics, just collect traces and infer metrics from them.

As for providing forms of projection, I think it would be no different than actual metrics.

As for requiring code change, yes that's right, the user application should be instrumented to collect/export the tracing data. Though I believe they could technically be inferred from the tracing add-on's emitted data (not 100% sure).

It's probably not applicable in the context of Linkerd, but did want to get the team's thoughts (if any) on it. :)

[ihcsim]

If the collected traces can only infer a subset of the Linkerd metrics, implying that Linkerd still need to collect other metrics, then I am not sure how much we are gaining. Any idea how one would test/validate the correctness of the inferred metrics? Sounds like a difficult thing to debug?

[naseemkullah]

Good points!
What is and example of the uninferrable linkerd metric?
As for test/validation I don't see how without having the actual metrics to compare to, I guess it's just faith in statistics. :p

[ihcsim]

What is and example of the uninferrable linkerd metric?

You mentioned total counts etc. Or did you mean the inference process is expensive to collect and store in general?

[naseemkullah]

Right I think a total count could be inferred from the count of spans and the sampling rate. But this inference process would first have to be made available in Jaeger or something outside of Linkerd. This whole trace inferred metric concept is very new to me and still confusing. Thanks for good points @ihcsim.

[grampelberg]

Isn't this a feature of the collector/backend for the trace implementation being used? Trying to infer metrics from traces seems like it'd just take more processing and introduce more complexity while reducing the fidelity of existing metrics.

[naseemkullah]

Isn't this a feature of the collector/backend for the trace implementation being used?

That sounds right.

Trying to infer metrics from traces seems like it'd just take more processing and introduce more complexity while reducing the fidelity of existing metrics.

True!