Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Symbolization Fails - Flame Graph Profile Lacks Meaningful function names #293

Open
JLowe-N opened this issue Jan 4, 2025 · 2 comments

Comments

@JLowe-N
Copy link

JLowe-N commented Jan 4, 2025

I am running:
otel-ebpf-profiler with the recommended Dockerfile, built today:

FROM ubuntu:22.04 as builder
RUN apt-get update && apt-get -y install wget gcc
RUN wget https://go.dev/dl/go1.22.10.linux-amd64.tar.gz
RUN tar -C /usr/local -xzf go1.22.10.linux-amd64.tar.gz
# fixed versions for pyroscope, otel-collector, otel-profiler due to protocol changes
RUN wget https://github.com/open-telemetry/opentelemetry-ebpf-profiler/archive/9eb454541a3cf1282a8755dbcec9a9471b243e83.tar.gz
RUN mkdir /profiler
RUN tar --strip-components=1 -C /profiler -xzf 9eb454541a3cf1282a8755dbcec9a9471b243e83.tar.gz
WORKDIR /profiler
RUN /usr/local/go/bin/go build .

FROM ubuntu:22.04

RUN apt-get update && \
    apt-get install -y linux-headers-generic && \
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /profiler/ebpf-profiler /usr/local/bin/

ENTRYPOINT ["/usr/local/bin/ebpf-profiler"]

This feeds into opentelemetry-collector-contrib:0.114.0 via OTLP.
My OTEL Collector then exports the profile signal over OTLP to Grafana Pyroscope (grafana/pyroscope:weekly-f96-1886f2c42).

I can see profile information coming into Grafana, but it lacks meaningful names and line numbers for my Python code.
Even when running a simple script in a loop, I fail to get meaningful information.
I have also tried standalone containers for python:3.9, python:3.11, and python:3.13, and have not had luck with getting useful profiles.
image

image

When comparing this to the example flamegraph, I see the example has the expected function names and line numbers.
image

Any thoughts on why I am not getting useful function names or line numbers for my code and/or Python libraries?
This application is running in Docker Compose, on an Azure VM Ubuntu 20.04.4 VM running Docker Engine 20.10.17.

@JLowe-N
Copy link
Author

JLowe-N commented Jan 6, 2025

Looks like I can get it mostly working in Devfiler, but not Pyroscope yet.
image

@florianl
Copy link
Contributor

florianl commented Jan 6, 2025

The described issue sounds similar to #285.
The OTel Profiling protocol is still in development and changes, including breaking changes not not following semantic conventions. And it seems that this repository adapts faster to these protocol changes than Grafana Pyroscope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants