Document how to define custom OTel resource attributes on exported telemetry

[beeme1mr]

Extend the OTel documentation to mention how custom resource attributes can be defined. Please see the following thread for more information.

Hey @apodgorbunschih, I talked to one of the mainters of OpenTelemetry and they think OTEL_RESOURCE_ATTRIBUTES is still the way to go if the attributes are static. The reason you didn't see it in your test was that you need to explicitly enable resources as labels in your Opentelemetry collector config.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter

The easiest way to do this is to enable resource_to_telemetry_conversion like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    // This is the new setting
    resource_to_telemetry_conversion:
      enabled: true
  otlp/jaeger:
    endpoint: jaeger-all-in-one:4317
    tls:
      insecure: true
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ otlp/jaeger ]
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ prometheus ]

However, this may not be a good idea if there are a large number of resources. The recommended approach is to apply a processor that extracts specific resource attributes as Prometheus labels.

processor:
  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
        - set(attributes["container"], resource.attributes["k8s.container.name"])
        - set(attributes["pod"], resource.attributes["k8s.pod.name"])

You can read more about it here.

Hope this helps!

Originally posted by @beeme1mr in #1462