gRPC load balancing

[Rathil]

Can't make Load balancer work when working with gRPC.

I have 2 services: router and project.
router - is a microservice that takes requests from outside Kuber and proxies them to a specific microservice (it's kind of like Ingress). It also translates requests from HTTPS to HTTP. All requests within Kube are made by unencrypted requests (HTTP not HTTPS).
project - one of the microservices that receives requests from router.

router:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: router
spec:
  selector:
    matchLabels:
      app: router
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: router
    spec:
      imagePullSecrets:
        - name: git-lab-token
      containers:
        - name: app
          image: XXX/ms/router:latest
          imagePullPolicy: Always
          args:
            - run
            - --db-mongo-db-name=XXX
            - --domain-suffix=XXX
            - --cert-email=XXX
          env:
            - name: DB_MONGO_URI
              valueFrom:
                secretKeyRef:
                  name: mongo-router
                  key: uri
---
apiVersion: v1
kind: Service
metadata:
  name: router
  labels:
    app: router
spec:
  type: LoadBalancer
  selector:
    app: router
  ports:
    - port: 80
      name: http
    - port: 443
      name: https

project:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: project
spec:
  selector:
    matchLabels:
      app: project
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: project
    spec:
      imagePullSecrets:
        - name: git-lab-token
      containers:
        - name: app
          image: XXX/ms/project:latest
          imagePullPolicy: Always
          args:
            - run
            - --db-mongo-db-name=XXX
          env:
            - name: DB_MONGO_URI
              valueFrom:
                secretKeyRef:
                  name: mongo-access
                  key: uri
---
apiVersion: v1
kind: Service
metadata:
  name: project-grpc
  labels:
    app: project
spec:
  selector:
    app: project
  ports:
    - port: 8081
      name: grpc

From the router I open the connection to the server side (and do not close the connection), but send separate gRPC requests (not stream) from the client side to the server side.

code:

// .....
conn, err := net.DialTimeout("tcp", serviceName+":8081", 3*time.Second)
if err != nil {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// TODO log => err
		w.WriteHeader(http.StatusBadGateway)
	})
}

proxy = &httputil.ReverseProxy{
	Director: func(r *http.Request) {
		r.URL.Scheme = "http"
		r.URL.Host = serviceName
		if ua, ok := r.Header["User-Agent"]; !ok || len(ua) == 0 {
			r.Header.Set("User-Agent", serverProxyDefaultUserAgent)
		}
	},
	Transport: &http2.Transport{
		AllowHTTP: true,
		DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) {
			return conn, nil
		},
	},
	FlushInterval: -1,
	ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
		// TODO log => err
		w.WriteHeader(http.StatusBadGateway)
	},
}
// ....
proxy.ServeHTTP(w,r)

Client code:

conn, err := grpc.Dial(
	"public.domain:443",
	grpc.WithTransportCredentials(credentials.NewTLS(nil)),
)
if err != nil {
	panic(err)
}
project := sdk.NewSProjectClient(conn)
for i := 0; i < 10000; i++ {
	_, _ = project.List(c, &sdk.List_Request{
		Limit: 5,
	})
	if i%100 == 0 {
		fmt.Println(i)
	}
}

Protocol:

syntax = "proto3";

package project.grpc.v1;

option go_package = ".;sdk";

service SProject {
    // Get list of projects
    rpc List (List.Request) returns (List.Response) {
    }
}

message List {
    message Request {
        int64 Limit = 1;
        int64 Offset = 2;
    }
    message Response {
        repeated Project Project = 1;
        bool HasNext = 2;
    }
}

message Project {
    string Id = 1;
    string Name = 2;
    int64 CreatedAt = 3;
}

It's like this on a dashboard:

[grampelberg]

What's the Host header? Check out the ingress documentation for some extra details around what's happening there.

[Rathil]

You're absolutely right! I just didn't put in the necessary HTTP headers!

[Rathil]

The solution specifically to my problem is adding a header to the proxying query:

r.Header.Set("l5d-dst-override", serviceName)

[haji4ref]

I have same issue!
Should we change our client side code? I thought there is no need to do these stuffs when we use some service mesh like linkerd!
@Rathil

[alpeb]

Are you also implementing a reverse proxy? I think the best would be to open a new discussion with some details specific to your scenario.