Enhance Kafka exporter to respect max message size

[yurishkuro]

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

The exporter has a config option MaxMessageBytes but it itself does not respect it and attempts to send a full serialized payload to the Kafka driver which may reject it based on this setting.

Describe the solution you'd like

Most payloads can be safely split into chunks of "safe" size that will be accepted by the Kafka driver. For example, in Jaeger integration tests there is a test that writes a trace with 10k spans, which is 3Mb in size when serialized as JSON. The trace can be trivially split into multiple messages that would fit in the default 1Mb size limit.

Describe alternatives you've considered

No response

Additional context

jaegertracing/jaeger#6437 (comment)

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[RenuBhati]

Hey @yurishkuro, I would like to contribute to this issue. Can you please assign me this?

[RenuBhati]

Hey @yurishkuro, I’m tied up with another project and it’s taking longer than expected. I won't be able to pick this up. Thanks for your understanding.

[LZiHaN]

hi @yurishkuro , is this task still available? I’d like to give it a try, assign please

[chahatsagarmain]

@LZiHaN are you working on this ?

[LZiHaN]

@LZiHaN are you working on this ?

Yes, I'm working on it.