Elastic /_bulk API response missing items array

[awesense-paul]

The elastic bulk API HTTP response contains an array field "items" (which includes the full documents that were inserted), but qryn does not include that field.

(I am using rsyslog w/omelasticsearch output module, and it logs a warning that the items[] array is missing. For my purposes, the response only needs to contain an empty array to avoid the warning)

[lmangani]

Is this a recent requirement? I do not recall this being mandatory for clients inserting data

[awesense-paul]

It appears in the the 7.17 docs (maybe earlier versions too? that's just the oldest I could find on the Elastic site)

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-bulk.html#bulk-api-response-body

I'm not sure how many clients actually require it. rsyslog seems to be fine with just an empty array. tbh it seems... kinda wasteful to have an API returning [aspects/portions of] the data that was in the request anyway? For my use-case, I'm just trying to get rsyslog to complain less.

[lmangani]

Indeed this sounds like as horrible as anything Elastic. My personal take is returning an empty array would be worse than returning nothing as it would suggest no action was performed potentially leading the client into repeating the bulk insertion. Out of curiosity have you considered just using one of the other log insertion methods without such silly requirements? The results are the same when querying data back.

[awesense-paul]

That is a good point w.r.t. returning empty array.

I do wonder whether I could get logs going into a metrico/otel-collector via a different protocol, that might actually handle the load better anyway. I'll try that instead (though this issue may still remain valid, just less important)

[lmangani]

@awesense-paul IMHO vector does fantastic with qryn logs and its one of the lighter agents out there

[awesense-paul]

Taking a look! Can it insert directly into CH? Or do you typically use the Loki sink (or other qryn-supported protocol)?

[lmangani]

For most agents using a loki sink with bulking works great - using direct CH ingestion really only makes a noticeable difference at huge volumes but I'm curious to know what you end up choosing and happy to assist if we can