gRPC through Traefik || Envoy
My weekend’s fun resulted in a gRPC client and server. I’ve spent some time familiarizing myself with Istio but I’d not spent any time directly with Envoy. Istio always ‘just works’ but I wanted to better understand Envoy and — having explored Traefik recently too (Golang-goodness) — I thought I’d try jamming these proxies between my client and server to better understand gRPC load-balancing. I bring you, gRPC three-ways: “French Vanilla” Traefik (containerized and not) and “Cookie Dough” Envoy.
TLS
Let’s do this properly and secure client →proxy and proxy →server. I’m confident there’s an easy way to do this with Let’s Encrypt but, for simplicity:
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/backend.key \
-out ./certs/backend.crt \
-subj "/CN=backend.local/O=backend.local"openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./certs/frontend.key \
-out ./certs/frontend.crt \
-subj "/CN=frontend.local/O=frontend.local"
NB Please do put the certs in a subdirectory (called
certs
) as this will facilitate using the Traefik and Envoy containers.
To resolve frontend.local
and backend.local
, you can (temporarily!) revise the localhost’s /etc/hosts
with:
127.0.0.1 frontend.local
127.0.0.1 backend.local
127.0.0.1 localhost
And, to confirm, ping ’em and ensure they resolve.
Traefik
It’s not cheating when the docs are good. If you follow this, you’ll have a working solution:
https://docs.traefik.io/user-guide/grpc/
Thanks to the Traefik docs folks, I got Traefik working *really* quickly.
Assuming you used the cert names as above, create traefik.docker.toml
and traefik.local.toml
(!) using the following. The difference between the two config files is that, it’s easier to map the the certs directory to the container’s root but not so easy to assume this for a local config. For Docker, use the toml
as below. For local, replace /certs/
with ./certs/
Then, either download the latest Traefik release and:
./traefik_linux-amd64 --configfile=./traefik.local.toml
NB I’m running on a 64-bit Linux, hence
-linux-amd64
. Pick the correct release for your OS.
Or, run the Traefik team’s latest container image:
docker run \
--interactive \
--rm \
--name=traefik \
--net=host \
--publish=127.0.0.1:80:80 \
--publish=127.0.0.1:8080:8080 \
--publish=127.0.0.1:4443:4443 \
--volume=$PWD/certs:/certs \
--volume=$PWD/traefik.docker.toml:/etc/traefik/traefik.toml \
--volume=/var/run/docker.sock:/var/run/docker.sock \
traefik --api --docker
NB We may not need
80
but8080
is for Traefik’s UI and4443
is the port that proxies our gRPC service.
Doesn’t do much (yet) ;-) Though you can check its UI:
http://localhost:8080/dashboard/#/
Using my gRPC client and server, we need to reconfigure both to use the frontend and backend certs/keys that we created previously and used to configure Traefik. Rather than replay the client and server in their entirety, something like:
client main.go:
const certPath = "${YOUR_CERT_PATH}"
frontendCert, _ := ioutil.ReadFile(
fmt.Sprintf("%v/frontend.crt", certPath))
roots := x509.NewCertPool()
roots.AppendCertsFromPEM(frontendCert)
credsClient := credentials.NewClientTLSFromCert(roots, "")...serverutils.Serve(
func(
request *serverutils.ProbeRequest,
reply *serverutils.ProbeReply) { conn, err := grpc.Dial(
fmt.Sprintf("%s:%s", address, port),
//grpc.WithInsecure()
grpc.WithTransportCredentials(credsClient))
...})
NB Don’t forget to (1) replace
${YOUR_CERT_PATH}
with the path to the cert you created previously and referenced in the Traefik configuration — the cert and not the key is requires by the client; (2) replacegrpc.WithInsecure()
withgrpc.WithTransportCredentials(credsClient)
.
server main.go:
const certPath = "${YOUR_CERT_PATH}"
backendCert, _ := ioutil.ReadFile(
fmt.Sprintf("%v/backend.crt", certPath))
backendKey, _ := ioutil.ReadFile(
fmt.Sprintf("%v/backend.key", certPath))
cert, err := tls.X509KeyPair(backendCert, backendKey)
if err != nil {
log.Fatalf("Failed to parse certificate: %v", err)
}
creds := credentials.NewServerTLSFromCert(&cert)
serverOption := grpc.Creds(creds)...s := grpc.NewServer(serverOption)
NB Don’t forget to replace
${YOUR_CERT_PATH}
with the path to the certs you created previously and referenced in the Traefik configuration; (2) refinegrpc.NewServer(serverOption)
.
While Traefik is running, start the server:
PROBE_PORT=50051 \
go run main.go
NB PROBE_PORT=50051
is redundant as this is the default. The server doesn’t know it’s own hostname but the proxy routes to backend.local
and, as you’ll recall, this defaults to localhost where we’re running the server.
Then start the client:
PROBE_HOST=frontend.local \
PROBE_PORT=4443 \
cloudprober --config_file=./cloudprober.local.cfg --logtostderr
NB We must explicitly set the client’s port because, not only do we want it to route to the proxy siting on
4443
instead of the server’s port of50051
(the proxy talks to the server) but the server won’t respond to the client because the client provide the proxy’s cert (identified asfrontend.local
). The client must referencefrontend.local
because this is the hostname of the proxy (albeit otherwise equivalent to localhost).
All being well, you should see probe requests and replies and you should be able to monitor metrics from the probe on :9313/metrics
as before. This time, you’ll also be able to observe Traefik’s metrics:
When you’re done, terminate the gRPC client and the Traefik proxy. You may leave the gRPC server running.
Envoy
I tried Googling “Envoy gRPC” to find a documented solution à la Traefik but was not successful. I found a sample config by guodongx
on a GitHub issue and I used this as the basis for munging what I had working with Traefik into a working solution for Envoy. Thanks guodongx!
Traefik and Envoy both perpetuate an excellent behavior in providing solutions distributed as single, standalone binaries. Envoy also biases towards using the containerized distribution. So, we’ll use that.
We first need a configuration… Ah, configuration…
NB This Envoy configuration reuses the certs and shares the same fundamental configuration with the Traefik configuration. The only difference is that Envoy expects PEM-formatted certs and keys, so, in the
./certs
directory:
openssl x509 -in backend.crt -out backend_crt.pem -outform PEM
openssl x509 -in frontend.crt -out frontend_crt.pem -outform PEMopenssl rsa -in backend.key -outform PEM -out backend_key.pem
openssl rsa -in frontend.key -outform PEM -out frontend_key.pem
And then, assuming you’ve terminated the Traefik proxy which shares the ports, you can run Envoy:
docker run \
--interactive \
--rm \
--name=envoy \
--net=host \
--publish=127.0.0.1:9901:9901 \
--publish=127.0.0.1:4443:4443 \
--volume=$PWD/envoy.yaml:/etc/envoy/envoy.yaml \
--volume=$PWD/certs:/certs/ envoyproxy/envoy:aa61c6c34f7bd4c1448649686b5bd7511aaa8d51
Once Envoy’s running, if you made the client and server changes to reflect the certs/keys already, you may start the client:
PROBE_HOST=frontend.local \
PROBE_PORT=4443 \
cloudprober --config_file=./cloudprober.local.cfg --logtostderr
And you should see probe requests and replies and be able to observe the probe (proxy) on :9313
and you should be able to interact with Envoy through it’s UI:
http://localhost:9901
You’ll note that Envoy exports Prometheus metrics too
http://localhost:9910/stats/prometheus
I’ll leave you to explore the other facilities.
When you’re done, terminate the gRPC client, the proxy and the gRPC server.
Conclusion
We’ve taken an (arbitrary) gRPC client|server pair and decoupled them using Traefik and Envoy. I’ll leave it as exercise for us all to scale to multiple gRPC clients and servers.
Aside: Kubernetes Namespaces
TL;DR Use them!
My colleague — Mike — and I wrote about the benefits/uses of Kubernetes Namespaces a couple of years ago. You’d think I’d have been a more committed user of Namespaces with my Kubernetes since. I’m lazy though and I tend to create clusters for specific projects and then whack ’em. However, even for the simplest scenarios — where I have say 2, distinct Kubernetes Deployments, Namespaces make sense.
Here’s one assailable awesome benefit of Namespaces, when you’re done:
kubectl delete namespace/delete-me
Sure, it’s a little rm -rf
but (!) it whacks all the resources in the namespace and provides a clean slate. So, even for my simple use-cases, I‘m going to be more diligent in creating Namespaces.