Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow tracing sampleRate smaller than 1% #5089

Open
dghubble opened this issue Jan 18, 2025 · 7 comments · May be fixed by #5125
Open

Allow tracing sampleRate smaller than 1% #5089

dghubble opened this issue Jan 18, 2025 · 7 comments · May be fixed by #5125
Labels
area/api API-related issues

Comments

@dghubble
Copy link

dghubble commented Jan 18, 2025

Description:

Envoy Gateway's EnvoyProxy CustomResourceDefinition expects the tracing field's sampleRate to be an integer, allowing 0 to 100.

I'd like to keep tracing on, but sample perhaps 0.01% of traffic. This does not seem to be directly possible, although envoy-proxy itself seems to support float64 sample rates.

# desired
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
spec:
  telemetry:
    tracing:
      sampleRate: 0.01
...

[optional Relevant Links:]

https://gateway.envoyproxy.io/latest/api/extension_types/#proxytelemetry

SamplingRate controls the rate at which traffic will be
selected for tracing if no prior sampling decision has been made.
Defaults to 100, valid values [0-100]. 100 indicates 100% sampling

@arkodg
Copy link
Contributor

arkodg commented Jan 18, 2025

cc @zirain , we may have been trying to avoid using floats in the CRD in the past, but this is a legit use case, and recently the faultInjection API used floats

Percentage *float32 `json:"percentage,omitempty"`

@zirain
Copy link
Member

zirain commented Jan 18, 2025

TBH, is that real meaningful in real world?

@arkodg
Copy link
Contributor

arkodg commented Jan 18, 2025

gateway-api's mirroring filter introduced an additional fraction field to solve this https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.HTTPRequestMirrorFilter, we could have something similar here

@dghubble
Copy link
Author

is that real meaningful in real world?

With a lot of traffic, 1% is a ton of traces. A fraction would help because even with 0.01% is doesn't take long to find traces with errors or p99s worth looking into.

@zirain
Copy link
Member

zirain commented Jan 19, 2025

you may not get help from traing with a very low sample rate, you may need CEL based sampler.
With CEL based sampler, you're able to sample only response.code != 200.

@dghubble
Copy link
Author

Im just looking to sample a smaller percentage of all traffic, not do expressions. Success traces still have value to me too

@zirain zirain added area/api API-related issues and removed triage labels Jan 19, 2025
@zirain
Copy link
Member

zirain commented Jan 19, 2025

sgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api API-related issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants