Enabling tracing by usage of opentelemetry

[rem1-dev]

Enabling tracing by usage of opentelemetry

[rem1-dev]

@shesek this is same as PR #99 but applied to fresh branch created out of new-index. Commits were squashed.

[rem1-dev]

I'm working on solving those conflict files

[rem1-dev]

@shesek conflicts resolved, it's ready to be merged.

[philippem]

@rem1-dev when testing locally we ran into some errors

tempo-1           | level=warn ts=2024-11-13T20:09:28.400374334Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185917 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"
tempo-1           | level=warn ts=2024-11-13T20:09:28.426261953Z caller=instance.go:42 msg="TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 185912 bytes to trace 6bafc5e34f7227d985e36a2142d95fb6 for tenant single-tenant"

which may be related to cardinality or size of the spans.

Possible causes:

- Large Trace Data: The trace contains too many spans or overly large data (such as logs or metadata).
- High Cardinality: Excessive number of unique tags, attributes, or data points being attached to spans, leading to large trace size.
- Excessive Span Detail: Individual spans may contain large payloads or detailed information that increase the trace size.
- Improper Sampling Rate: Sampling configuration might be too high, causing too many traces to be captured, some of which may exceed the size limit.
- Large Payloads in Spans: If spans contain large amounts of data (e.g., large JSON bodies, files, or detailed logs), it could push the trace size over the limit.
- Too Many Traces Collected: An overly aggressive tracing configuration that collects an excessive number of traces, increasing the likelihood of exceeding the size limit.
- Long or Complex Transactions: If a trace represents a very large or complex operation (e.g., a multi-step workflow or a distributed operation), it might naturally grow too large.

[rem1-dev]

@philippem how can I reproduce this locally?

[philippem]

@shesek what do you think about adding a new macro to reduce the amount of copy-paste? will require new module

[rem1-dev]

I provided a macro that wraps long #[tracing::instrument(args...)] macro call into simplified #[instrumented] call with optional capturing of instrumented function arguments. @philippem @shesek please take a look.

[shesek]

Looks good! I like the new macro :)

Now that we have it, perhaps we could also implement my suggestion from the prior PR to expand #[trace] into a no-op when tracing is disabled?

That way we won't need the negative no-otlp-tracing feature, can remove the manual compile_errors for invalid feature combos, and avoid the dependency on the tracing crate when tracing is disabled.

[rem1-dev]

I'm glad new macro looks good. I looked at possibility of removal of negative feature flag: no-otlp-tracing and I think it's still not feasible (correct me if I'm wrong). I think so because tracing requires main function to be:

• async
• have tokio::main attribute
• initialise tracing

so whether above is applied to main or not is controlled by no-otlp-tracing and otlp-tracing feature flags

[shesek]

so whether above is applied to main or not is controlled by no-otlp-tracing and otlp-tracing feature flags

Since we no longer need any dependencies for the "no otlp" case, the main() function can instead use #[cfg(not(feature = "otlp-tracing"))]

[shesek]

The refactoring looks great! 👍

Added a couple minor comments.

[philippem]

TACK 4421d

[philippem]

TACK 4421d1f

[rem1-dev]

I'm not sure why CI failed ... it suggests setting version for home crate:

cargo update [email protected] --precise ver
where `ver` is the latest version of `home` supporting rustc 1.75.0

I haven't introduced home in this PR. It's a dependency for electrumd and bitcoind.

On the other hand at some point I regenerated Cargo.lock because I was not able to handle all merge conflict. That's when home version was bumped from 0.5.9 to 0.5.11 so I'll try doing:

cargo update [email protected] --precise 0.5.9