Protocol error when using gRPC

[yangyulely]

What happened?

Steps to Reproduce

1. We are using node auto-instrumentations to collect telemetry data and send to opentelemetry collector.
2. Our collector was deployed on AWS EKS, and create a nginx ingress to expose collecting API.
3. We try to connect to opentelemetry collector with gRPC protocol.

Expected Result

Collect telemetry data and send to opentelemetry collector.

Actual Result

Protocol error when connecting to collector endpoint.

Additional Details

Java auto instrumentations works fine with gRPC, but nodejs not.
Opentelemetry colector ingress yaml as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: otel-ingress
  namespace: otel
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  tls:
    - hosts:
        - otel.example.com
      secretName: otel-tls-secret
  ingressClassName: nginx
  rules:
    - host: otel.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: otel-collector-opentelemetry-collector
                port:
                  number: 4317

OpenTelemetry Setup Code

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel.example.com"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_INSECURE="true"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
export OTEL_SERVICE_NAME="grpc_nodejs_demo"
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
cd $HOME/grpc_nodejs_demo
node $HOME/grpc_nodejs_demo/index.js

package.json

{
  "name": "nodejs_demo",
  "version": "1.0.0",
  "description": "nodejs demo for otel collector",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "license": "ISC",
  "dependencies": {
    "@grpc/grpc-js": "^1.12.5",
    "@opentelemetry/api": "^1.7.0",
    "@opentelemetry/auto-instrumentations-node": "^0.40.3",
    "@opentelemetry/exporter-logs-otlp-proto": "^0.49.1",
    "@opentelemetry/sdk-logs": "^0.49.1",
    "@opentelemetry/sdk-node": "^0.49.1",
    "body-parser": "^1.20.2",
    "cors": "^2.8.5",
    "express": "^4.18.2",
    "node-fetch": "2.6.2"
  }
}

Relevant log output

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error
    at callErrorFromStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
    at ServiceClientImpl.export (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at OTLPTraceExporter.send [as _send] (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/util.js:76:39)
    at /home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:44:18
    at new Promise (<anonymous>)
    at OTLPTraceExporter._sendPromise (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:43:25)
    at Immediate.<anonymous> (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:74:22)
    at process.processImmediate (node:internal/timers:478:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 13,
  details: 'Received RST_STREAM with code 2 triggered by internal client error: Protocol error',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Operating System and Version

Ubuntu 22.04

Runtime and Version

Node.js v20.11.0

[dudo]

I ran into something similar, and I ended up having to remove the protocol from the url

export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel.example.com"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

There's a mismatch there. Try

export OTEL_EXPORTER_OTLP_ENDPOINT="otel.example.com"

[pichlermarc]

@yangyulely thanks for reaching out. Can you try updating to the latest version? I'm not able to reproduce this with the latest release. Also: please specify the port - from what I know the speciation does not mention anything about which port to use so the grpc exporters default to what @grpc/gprc-js would use (which is port 443)

[yangyulely]

@dudo @pichlermarc, Thanks for replying, it is solved when I change collector endpoint to https://otel.example.com. I think http use 80 port, https use 443. Protocol must be https when using @grpc/gprc-js. Well, java agent no problem, probably because there's no this check.