Grpc load balancing not working

[c3dric373]

Hello,
I followed the instructions in the Getting Started documentation but somehow the grpc load balancing does not seem to work. See the attached screenshot, one voting service is receiving all the requests. I also attached the yaml file used to deploy the meshed services. Does anyone have an idea how I could debug this?

Here is the yaml file used to deploy:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    linkerd.io/inject: enabled
  name: emojivoto
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: emoji
  namespace: emojivoto
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: voting
  namespace: emojivoto
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: web
  namespace: emojivoto
---
apiVersion: v1
kind: Service
metadata:
  name: emoji-svc
  namespace: emojivoto
spec:
  ports:
  - name: grpc
    port: 8080
    targetPort: 8080
  - name: prom
    port: 8801
    targetPort: 8801
  selector:
    app: emoji-svc
---
apiVersion: v1
kind: Service
metadata:
  name: voting-svc
  namespace: emojivoto
spec:
  ports:
  - name: grpc
    port: 8080
    targetPort: 8080
  - name: prom
    port: 8801
    targetPort: 8801
  selector:
    app: voting-svc
---
apiVersion: v1
kind: Service
metadata:
  name: web-svc
  namespace: emojivoto
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: web-svc
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: emoji
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/version: v11
  name: emoji
  namespace: emojivoto
spec:
  replicas: 1
  selector:
    matchLabels:
      app: emoji-svc
      version: v11
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: emoji-svc
        version: v11
    spec:
      containers:
      - env:
        - name: GRPC_PORT
          value: "8080"
        - name: PROM_PORT
          value: "8801"
        image: docker.l5d.io/buoyantio/emojivoto-emoji-svc:v11
        name: emoji-svc
        ports:
        - containerPort: 8080
          name: grpc
        - containerPort: 8801
          name: prom
        resources:
          requests:
            cpu: 100m
      serviceAccountName: emoji
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: vote-bot
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/version: v11
  name: vote-bot
  namespace: emojivoto
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vote-bot
      version: v11
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: vote-bot
        version: v11
    spec:
      containers:
      - command:
        - emojivoto-vote-bot
        env:
        - name: WEB_HOST
          value: web-svc.emojivoto:80
        image: docker.l5d.io/buoyantio/emojivoto-web:v11
        name: vote-bot
        resources:
          requests:
            cpu: 10m
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: voting
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/version: v11
  name: voting
  namespace: emojivoto
spec:
  replicas: 2
  selector:
    matchLabels:
      app: voting-svc
      version: v11
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: voting-svc
        version: v11
    spec:
      containers:
      - env:
        - name: GRPC_PORT
          value: "8080"
        - name: PROM_PORT
          value: "8801"
        image: docker.l5d.io/buoyantio/emojivoto-voting-svc:v11
        name: voting-svc
        ports:
        - containerPort: 8080
          name: grpc
        - containerPort: 8801
          name: prom
        resources:
          requests:
            cpu: 100m
      serviceAccountName: voting
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/version: v11
  name: web
  namespace: emojivoto
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-svc
      version: v11
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: web-svc
        version: v11
    spec:
      containers:
      - env:
        - name: WEB_PORT
          value: "8080"
        - name: EMOJISVC_HOST
          value: emoji-svc.emojivoto:8080
        - name: VOTINGSVC_HOST
          value: voting-svc.emojivoto:8080
        - name: INDEX_BUNDLE
          value: dist/index_bundle.js
        image: docker.l5d.io/buoyantio/emojivoto-web:v11
        name: web-svc
        ports:
        - containerPort: 8080
          name: http
        resources:
          requests:
            cpu: 100m
      serviceAccountName: web
---

[aravind-vis]

@c3dric373 - were you able to get this working? I am followed the same doc and don't find the load being balanced.

[georgkoester]

Dear @c3dric373 , did you make progress?

I believe your question is not a 100% correct: There are 0.3 RPS reaching one service, 1.3 RPS the first service. Response times are similar.
I could imagine the EWMA algorithm not needing to rebalance (yet).

This algorithm is somewhat arcane to me, that's why I am crawling the forum. I am investigating rebalancing issues experienced during rapid scale out on our side and would be interested in any further experience on your side.

[wmorgan]

Linkerd's EWMA algorithm does not guarantee an even distribution across pods, just low latency. It is happy to send all requests to one pod if that pod can handle the load. If you add more requests you should see a closer-to-even load.

[georgkoester]

Thank you @wmorgan .
We will now try to avoid a stampede to give linkerd time to adjust balancing.
Our assumption is that the algorithm and/or service discovery does not quickly start to take the new services into account. They seem to be appearing faster as ready on prometheus than high on the linkerd priority list.