gRPC through Traefik || Envoy

My weekend’s fun resulted in a gRPC client and server. I’ve spent some time familiarizing myself with Istio but I’d not spent any time directly with Envoy. Istio always ‘just works’ but I wanted to better understand Envoy and — having explored Traefik recently too (Golang-goodness) — I thought I’d try jamming these proxies between my client and server to better understand gRPC load-balancing. I bring you, gRPC three-ways: “French Vanilla” Traefik (containerized and not) and “Cookie Dough” Envoy.

TLS

Let’s do this properly and secure client →proxy and proxy →server. I’m confident there’s an easy way to do this with Let’s Encrypt but, for simplicity:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/backend.key \
-out ./certs/backend.crt \
-subj "/CN=backend.local/O=backend.local"
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/frontend.key \
-out ./certs/frontend.crt \
-subj "/CN=frontend.local/O=frontend.local"

NB Please do put the certs in a subdirectory (called certs) as this will facilitate using the Traefik and Envoy containers.

To resolve frontend.local and backend.local, you can (temporarily!) revise the localhost’s /etc/hosts with:

127.0.0.1 frontend.local
127.0.0.1 backend.local
127.0.0.1 localhost

And, to confirm, ping ’em and ensure they resolve.

Traefik

It’s not cheating when the docs are good. If you follow this, you’ll have a working solution:

https://docs.traefik.io/user-guide/grpc/

Thanks to the Traefik docs folks, I got Traefik working *really* quickly.

Assuming you used the cert names as above, create traefik.docker.toml and traefik.local.toml (!) using the following. The difference between the two config files is that, it’s easier to map the the certs directory to the container’s root but not so easy to assume this for a local config. For Docker, use the toml as below. For local, replace /certs/ with ./certs/

Then, either download the latest Traefik release and:

./traefik_linux-amd64 --configfile=./traefik.local.toml

NB I’m running on a 64-bit Linux, hence -linux-amd64. Pick the correct release for your OS.

Or, run the Traefik team’s latest container image:

docker run \
--interactive \
--rm \
--name=traefik \
--net=host \
--publish=127.0.0.1:80:80 \
--publish=127.0.0.1:8080:8080 \
--publish=127.0.0.1:4443:4443 \
--volume=$PWD/certs:/certs \
--volume=$PWD/traefik.docker.toml:/etc/traefik/traefik.toml \
--volume=/var/run/docker.sock:/var/run/docker.sock \
traefik --api --docker

NB We may not need 80 but 8080 is for Traefik’s UI and 4443 is the port that proxies our gRPC service.

Doesn’t do much (yet) ;-) Though you can check its UI:

http://localhost:8080/dashboard/#/

Using my gRPC client and server, we need to reconfigure both to use the frontend and backend certs/keys that we created previously and used to configure Traefik. Rather than replay the client and server in their entirety, something like:

client main.go:

const certPath = "${YOUR_CERT_PATH}"
frontendCert, _ := ioutil.ReadFile(
  fmt.Sprintf("%v/frontend.crt", certPath))
roots := x509.NewCertPool()
roots.AppendCertsFromPEM(frontendCert)
credsClient := credentials.NewClientTLSFromCert(roots, "")
...
serverutils.Serve(
  func(
    request *serverutils.ProbeRequest,
    reply *serverutils.ProbeReply) {
  conn, err := grpc.Dial(
   fmt.Sprintf("%s:%s", address, port),
   //grpc.WithInsecure()
   grpc.WithTransportCredentials(credsClient))

...
})

NB Don’t forget to (1) replace ${YOUR_CERT_PATH} with the path to the cert you created previously and referenced in the Traefik configuration — the cert and not the key is requires by the client; (2) replace grpc.WithInsecure() with grpc.WithTransportCredentials(credsClient).

server main.go:

const certPath = "${YOUR_CERT_PATH}"
backendCert, _ := ioutil.ReadFile(
  fmt.Sprintf("%v/backend.crt", certPath))
backendKey, _ := ioutil.ReadFile(
  fmt.Sprintf("%v/backend.key", certPath))
cert, err := tls.X509KeyPair(backendCert, backendKey)
if err != nil {
  log.Fatalf("Failed to parse certificate: %v", err)
}
creds := credentials.NewServerTLSFromCert(&cert)
serverOption := grpc.Creds(creds)
...
s := grpc.NewServer(serverOption)

NB Don’t forget to replace ${YOUR_CERT_PATH} with the path to the certs you created previously and referenced in the Traefik configuration; (2) refine grpc.NewServer(serverOption).

While Traefik is running, start the server:

PROBE_PORT=50051 \
go run main.go

NB PROBE_PORT=50051 is redundant as this is the default. The server doesn’t know it’s own hostname but the proxy routes to backend.local and, as you’ll recall, this defaults to localhost where we’re running the server.

Then start the client:

PROBE_HOST=frontend.local \
PROBE_PORT=4443 \
cloudprober --config_file=./cloudprober.local.cfg --logtostderr

NB We must explicitly set the client’s port because, not only do we want it to route to the proxy siting on 4443 instead of the server’s port of 50051 (the proxy talks to the server) but the server won’t respond to the client because the client provide the proxy’s cert (identified asfrontend.local). The client must reference frontend.local because this is the hostname of the proxy (albeit otherwise equivalent to localhost).

All being well, you should see probe requests and replies and you should be able to monitor metrics from the probe on :9313/metrics as before. This time, you’ll also be able to observe Traefik’s metrics:

http://localhost:8080/metrics

Traefik’s (Prometheus) metrics endpoint

When you’re done, terminate the gRPC client and the Traefik proxy. You may leave the gRPC server running.

Envoy

I tried Googling “Envoy gRPC” to find a documented solution à la Traefik but was not successful. I found a sample config by guodongx on a GitHub issue and I used this as the basis for munging what I had working with Traefik into a working solution for Envoy. Thanks guodongx!

Traefik and Envoy both perpetuate an excellent behavior in providing solutions distributed as single, standalone binaries. Envoy also biases towards using the containerized distribution. So, we’ll use that.

We first need a configuration… Ah, configuration…

NB This Envoy configuration reuses the certs and shares the same fundamental configuration with the Traefik configuration. The only difference is that Envoy expects PEM-formatted certs and keys, so, in the ./certs directory:

openssl x509 -in backend.crt -out backend_crt.pem -outform PEM
openssl x509 -in frontend.crt -out frontend_crt.pem -outform PEM
openssl rsa -in backend.key -outform PEM -out backend_key.pem
openssl rsa -in frontend.key -outform PEM -out frontend_key.pem

And then, assuming you’ve terminated the Traefik proxy which shares the ports, you can run Envoy:

docker run \
--interactive \
--rm \
--name=envoy \
--net=host \
--publish=127.0.0.1:9901:9901 \
--publish=127.0.0.1:4443:4443 \
--volume=$PWD/envoy.yaml:/etc/envoy/envoy.yaml \
--volume=$PWD/certs:/certs/ envoyproxy/envoy:aa61c6c34f7bd4c1448649686b5bd7511aaa8d51

Once Envoy’s running, if you made the client and server changes to reflect the certs/keys already, you may start the client:

PROBE_HOST=frontend.local \
PROBE_PORT=4443 \
cloudprober --config_file=./cloudprober.local.cfg --logtostderr

And you should see probe requests and replies and be able to observe the probe (proxy) on :9313 and you should be able to interact with Envoy through it’s UI:

http://localhost:9901

Envoy’s fully-featured UI

You’ll note that Envoy exports Prometheus metrics too

http://localhost:9910/stats/prometheus

Envoy’s (Prometheus) stats endpoint

I’ll leave you to explore the other facilities.

When you’re done, terminate the gRPC client, the proxy and the gRPC server.

Conclusion

We’ve taken an (arbitrary) gRPC client|server pair and decoupled them using Traefik and Envoy. I’ll leave it as exercise for us all to scale to multiple gRPC clients and servers.

Aside: Kubernetes Namespaces

TL;DR Use them!

My colleague — Mike — and I wrote about the benefits/uses of Kubernetes Namespaces a couple of years ago. You’d think I’d have been a more committed user of Namespaces with my Kubernetes since. I’m lazy though and I tend to create clusters for specific projects and then whack ’em. However, even for the simplest scenarios — where I have say 2, distinct Kubernetes Deployments, Namespaces make sense.

Here’s one assailable awesome benefit of Namespaces, when you’re done:

kubectl delete namespace/delete-me

Sure, it’s a little rm -rf but (!) it whacks all the resources in the namespace and provides a clean slate. So, even for my simple use-cases, I‘m going to be more diligent in creating Namespaces.