Failed to scrape Prometheus endpoint

[oszlak]

Describe the bug
I'm setting up target allocator with Promethues receiver(deployed as daemonset) but getting this error from each otel agent:

2025-01-05T19:40:04.672Z warn internal/transaction.go:129 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1736106004668, "target_labels": "{__name__=\"up\", endpoint=\"https-metrics\", instance=\"10.244.15.227:10250\", job=\"kubelet\", metrics_path=\"/metrics/probes\", namespace=\"kube-system\", node=\"ip-10-244-15-227.ec2.internal\", service=\"isr-playground-k8s-cen-dev-kubelet\"}"}

also for up and kube-system
Steps to reproduce
Deploying otel collector as daemonset, with only promethues receiver and enabling target allocator

What did you expect to see?
The agent can scrape all the metrics
What did you see instead?
The error above

What version did you use?
v0.116.0

What config did you use?

# Name configuration
nameOverride: "otel-scraper"
fullnameOverride: "otel-scraper"

# Deployment mode
mode: daemonset
image:
  # If you want to use the core image `otel/opentelemetry-collector`, you also need to change `command.name` value to `otelcol`.
  repository: opentelemetry-collector
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "0.116.1"
command:
  name: "otelcol-k8s"

# Target Allocator configuration
targetAllocator:
  enabled: true
  prometheusCR:
    enabled: true
  allocationStrategy: "per-node"
  image:
    repository: opentelemetry-targerallocator
    tag: "0.116.0"
    pullPolicy: IfNotPresent
  replicas: 1
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 50m
      memory: 64Mi

# Main collector configuration
config:
  receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: "otel-collector"
            scrape_interval: 60s
            static_configs:
              - targets: ["0.0.0.0:8888"]
          - job_name: "kubelet"
            scheme: "https"
            tls_config:
              insecure_skip_verify: true
            kubernetes_sd_configs:
              - role: node
      target_allocator:
        endpoint: "http://otel-scraper-targetallocator:80"
        interval: 60s
        collector_id: "${POD_NAME}"

# Enable relevant ports
ports:
  prometheus:
    enabled: true
    containerPort: 8888
    servicePort: 8888
    protocol: TCP

# Set up environment variables for the POD_NAME
extraEnvs:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name

# Service account configuration
serviceAccount:
  create: true
  name: "otel-scraper"

# Basic resource limits
resources:
  limits:
    cpu: 1
    memory: 2Gi
  requests:
    cpu: 200m
    memory: 400Mi

# Pod annotations for Prometheus scraping
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8888"

# Service configuration
service:
  enabled: true
  type: ClusterIP
clusterRole:
  create: true

Environment
AMI on k8s

Additional context

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[dashpole]

Can you raise logging level to debug? That should print out the reason why the scrape failed.