OTLP/gRPC message size limit is fixed at 4MB

[carsonip]

If the decompressed grpc message exceeds 4MB, the collector sending to apm-server logs the following error.

2024-11-25T04:08:26.561Z error internal/queue_sender.go:100 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (6624956 vs. 4194304)", "dropped_items": 11613}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue_sender.go:100

The limit comes from the default 4MB grpc server message size limit MaxRecvMsgSize in https://pkg.go.dev/google.golang.org/grpc#MaxRecvMsgSize .

Current user workarounds

• if batchprocessor is used, adjust the config values to influence the payload size
• use otlphttpexporter in place of otlpexporter in the collector sending to apm-server.

Proposed solution

apm-server should either document or adjust the limit, or make it configurable.

[inge4pres]

Do you know if there's a corresponding config in OTel collectors?
Would it be sensible to mirror its name in apm-server?

[carsonip]

Do you know if there's a corresponding config in OTel collectors?

max_recv_msg_size_mib in https://github.com/open-telemetry/opentelemetry-collector/blob/33264a5408cd918cfb45fd622b99f681c67d535c/config/configgrpc/configgrpc.go#L175

Would it be sensible to mirror its name in apm-server?

If we're adding a config, it needs to be nested under an appropriate section, as it only concerns OTLP/gRPC, not even OTLP/HTTP. We don't have any otel specific sections in apm-server.yml.

[axw]

Maybe we should make it configurable, but IMO the better solution would be for clients to send smaller payloads. If the server can accept a larger payload, then it will be more susceptible to OOM.