[elasticsearchexporter]: Add dynamic document id support for logs #37065

mauri870 · 2025-01-07T13:45:52Z

Description

This PR adds a new config option logs_dynamic_id that when set to true reads the elasticsearch.document_id attribute from each log record and uses it as the final document id in Elasticsearch. This is only implemented for logs but I can open subsequent PRs supporting metrics and traces akin to the *_dynamic_index options.

Fixes #36882

Testing

Added tests to verify that the document ID attribute can be read from the log record and that the _id is properly forwarded to Elasticsearch. Also asserted that when there is no doc id attribute the current behavior is retained.

Documentation

Updated the readme to mention the new logs_dynamic_id config option.

exporter/elasticsearchexporter/README.md

carsonip

looks good at a high level, thanks!

.chloggen/elasticsearchexporter_logs_dynamic_id.yaml

exporter/elasticsearchexporter/exporter.go

carsonip

Q: do you expect the elasticsearch.document_id to be indexed under attributes as well, or should it be ignored during serialization?

mauri870 · 2025-01-09T14:57:40Z

Q: do you expect the elasticsearch.document_id to be indexed under attributes as well, or should it be ignored during serialization?

Haven't thought of that, but I think it makes sense to remove the field, as it only has meaning for the exporter.

Also, elasticsearch.document_id is overly specific, I'm happy to hear suggestions on the naming.

mauri870 · 2025-01-09T18:25:54Z

I updated the code to remove the id field from the final document.

Regarding the attribute name, I wonder if we should stick to some kind of semantic convention such as https://opentelemetry.io/docs/specs/semconv/database/elasticsearch/. I scrolled throught it but couldn't find an existing attribute for a document id.

carsonip · 2025-01-09T21:47:18Z

exporter/elasticsearchexporter/exporter.go

+func (e *elasticsearchExporter) extractDocumentIDAttribute(m pcommon.Map) string {
+	if e.config.LogsDynamicID.Enabled {
+		docID, ok := getFromAttributes(documentIDAttributeName, "", m)
+		m.Remove(documentIDAttributeName)
+		if docID != "" && ok {
+			return docID
+		}
+	}
+	return ""


Suggested change

func (e *elasticsearchExporter) extractDocumentIDAttribute(m pcommon.Map) string {

if e.config.LogsDynamicID.Enabled {

docID, ok := getFromAttributes(documentIDAttributeName, "", m)

m.Remove(documentIDAttributeName)

if docID != "" && ok {

return docID

}

}

return ""

func (e *elasticsearchExporter) extractDocumentIDAttribute(m pcommon.Map) (docID string) {

if e.config.LogsDynamicID.Enabled {

m.RemoveIf(func(k string, value pcommon.Value) bool {

if k == documentIDAttributeName {

docID = value.AsString()

return true

}

return false

})

}

return

}

nit: Get and Remove are both O(N) in pcommon.Map. Use RemoveIf to iterate over the map only once.

carsonip · 2025-01-09T21:59:33Z

exporter/elasticsearchexporter/config.go

@@ -61,6 +61,9 @@ type Config struct {
 	// fall back to pure TracesIndex, if 'elasticsearch.index.prefix' or 'elasticsearch.index.suffix' are not found in resource or attribute (prio: resource > attribute)
 	TracesDynamicIndex DynamicIndexSetting `mapstructure:"traces_dynamic_index"`

+	// LogsDynamicID is used to configure the document id for logs.
+	LogsDynamicID DynamicIndexSetting `mapstructure:"logs_dynamic_id"`


Suggested change

LogsDynamicID DynamicIndexSetting `mapstructure:"logs_dynamic_id"`

LogsDynamicID bool `mapstructure:"logs_dynamic_id"`

nit: This has nothing to do with Index, which makes the use of DynamicIndexSetting slightly odd.

Also, personally I find this 1 level of nesting not very ergonomic. Unless we plan to add features to dynamic ID in the future e.g. logs_dynamic_id::attribute_name to make the attribute name configurable, I feel like adding a layer just to have a ::enabled bool not very useful. You can tell I'm already not very happy with *_dynamic_index::enabled. But I'd be interested to know other codeowners' opinion on this.

carsonip · 2025-01-09T22:02:42Z

exporter/elasticsearchexporter/config.go

@@ -61,6 +61,9 @@ type Config struct {
 	// fall back to pure TracesIndex, if 'elasticsearch.index.prefix' or 'elasticsearch.index.suffix' are not found in resource or attribute (prio: resource > attribute)
 	TracesDynamicIndex DynamicIndexSetting `mapstructure:"traces_dynamic_index"`

+	// LogsDynamicID is used to configure the document id for logs.


Suggested change

// LogsDynamicID is used to configure the document id for logs.

// LogsDynamicID configures whether log record attribute `elasticsearch.document_id` is set as the document ID in ES.

carsonip · 2025-01-09T22:03:21Z

exporter/elasticsearchexporter/README.md

+- `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
+  - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is != "" in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch.


As discussed below. Also would be good to mention that the attribute would be stripped.

carsonip · 2025-01-09T22:05:45Z

.chloggen/elasticsearchexporter_logs_dynamic_id.yaml

+component: elasticsearchexporter
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Support for dynamically setting the document ID of log records.


Suggested change

note: Support for dynamically setting the document ID of log records.

note: Add config `logs_dynamic_id` to dynamically set the document ID of log records using log record attribute `elasticsearch.document_id`

mauri870 added 4 commits January 7, 2025 09:30

Add support for setting a document id

1c3b656

run tests in parallel, test with sync and async bulk indexer

7128be6

add changelog entry

c277b55

remove todo for metrics and traces

89f19d3

mauri870 requested a review from a team as a code owner January 7, 2025 13:45

mauri870 requested a review from djaglowski January 7, 2025 13:45

github-actions bot assigned ChrsMark Jan 7, 2025

github-actions bot added the exporter/elasticsearch label Jan 7, 2025

github-actions bot requested review from carsonip, JaredTan95 and lahsivjar January 7, 2025 13:46

mauri870 changed the title ~~[elasticsearchexporter]: Add support for for setting a document id for logs~~ [elasticsearchexporter]: Add dynamic document id support for logs Jan 7, 2025

Merge branch 'main' into elasticsearchexporter-id

bebd2e5

axw reviewed Jan 8, 2025

View reviewed changes

exporter/elasticsearchexporter/README.md Outdated Show resolved Hide resolved

mauri870 marked this pull request as draft January 8, 2025 11:10

only look into record attributes

dc25057

mauri870 marked this pull request as ready for review January 8, 2025 12:13

github-actions bot assigned dmitryax Jan 8, 2025

carsonip reviewed Jan 8, 2025

View reviewed changes

.chloggen/elasticsearchexporter_logs_dynamic_id.yaml Outdated Show resolved Hide resolved

exporter/elasticsearchexporter/exporter.go Outdated Show resolved Hide resolved

mauri870 added 2 commits January 9, 2025 08:35

fixes from code review

7f2557c

clarify about empty string value

1080790

carsonip reviewed Jan 9, 2025

View reviewed changes

mauri870 added 2 commits January 9, 2025 15:17

remove id attribute from the final document

9e0ad2f

Merge branch 'main' into elasticsearchexporter-id

d2349db

carsonip reviewed Jan 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[elasticsearchexporter]: Add dynamic document id support for logs #37065

[elasticsearchexporter]: Add dynamic document id support for logs #37065

mauri870 commented Jan 7, 2025 •

edited

Loading

carsonip left a comment

carsonip left a comment

mauri870 commented Jan 9, 2025 •

edited

Loading

mauri870 commented Jan 9, 2025

carsonip Jan 9, 2025

carsonip Jan 9, 2025

carsonip Jan 9, 2025

carsonip Jan 9, 2025

carsonip Jan 9, 2025

	LogsDynamicID DynamicIndexSetting `mapstructure:"logs_dynamic_id"`
	LogsDynamicID bool `mapstructure:"logs_dynamic_id"`

	// LogsDynamicID is used to configure the document id for logs.
	// LogsDynamicID configures whether log record attribute `elasticsearch.document_id` is set as the document ID in ES.

		- `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
		- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is != "" in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch.

	note: Support for dynamically setting the document ID of log records.
	note: Add config `logs_dynamic_id` to dynamically set the document ID of log records using log record attribute `elasticsearch.document_id`

[elasticsearchexporter]: Add dynamic document id support for logs #37065

Are you sure you want to change the base?

[elasticsearchexporter]: Add dynamic document id support for logs #37065

Conversation

mauri870 commented Jan 7, 2025 • edited Loading

Description

Testing

Documentation

carsonip left a comment

Choose a reason for hiding this comment

carsonip left a comment

Choose a reason for hiding this comment

mauri870 commented Jan 9, 2025 • edited Loading

mauri870 commented Jan 9, 2025

carsonip Jan 9, 2025

Choose a reason for hiding this comment

carsonip Jan 9, 2025

Choose a reason for hiding this comment

carsonip Jan 9, 2025

Choose a reason for hiding this comment

carsonip Jan 9, 2025

Choose a reason for hiding this comment

carsonip Jan 9, 2025

Choose a reason for hiding this comment

mauri870 commented Jan 7, 2025 •

edited

Loading

mauri870 commented Jan 9, 2025 •

edited

Loading