How to talk REST to containerd gRPC interface and more

11 min readAug 20, 2019

After successfully setting up my environment to have my browser communicate with containerd securely using gRPC-web and Envoy proxy as explained in my post Secure Browser Communication with Containerd Using gRPC, Envoy and OAuth 2.0 , I wanted to explore whether I could design and setup an architecture where not only Browser Javascript, but also any HTTP/REST client would be able to talk to containerd’s gRPC interface. So I wanted this environment to support gRPC clients, browsers and also any other legacy clients which need to stick to HTTP/REST protocol for a while.

I looked at different solutions around REST to gRPC, and I decided to give the gRPC Gateway project a try. As explained in the opening page of the project:

“ The grpc-gateway is a plugin of the Google protocol buffers compiler protoc. It reads protobuf service definitions and generates a reverse-proxy server which translates a RESTful JSON API into gRPC. This server is generated according to thegoogle.api.http annotations in your service definitions.”

This is a bit different from the Envoy gRPC-web filter solution. In that case, browser had the knowledge of the protobuf definitions, hence had the responsibility of translating between json body that Javascript understood and binary body that gRPC used. So browser had to know about the proto file and also we had to generate client side JS libraries for gRPC and protobuf. With the gRPC-gateway in the middle, the knowledge of translating between protobuf and JSON remains within the responsibility of grpc-gateway, hence client does not need to have a specific library, thus it can be any client that supports basic HTTP 1.1/REST. In this case, grpc-gateway acts as an ou-of-process reverse proxy that receives the REST request and forwards to gRPC server.

Many thanks to Brandon Philips for grpc-gateway-example that showed me a lot about using grpc-gateway. All files mentioned below (except for keys and certificates which any reader should generate using instructions below) can be found in my github repo: https://github.com/mark-kose/containerd-grpc-examples/tree/master/containerd-grpc-gateway

I have to warn you. This is an extensive reading, and requires focus and effort. And a lot of typing(or copy/pasting if you prefer)

gRPC-gateway setup

We start by downloading necessary packages. So let’s get gateway package first:

go get github.com/grpc-ecosystem/grpc-gateway/runtime

Now, during my testing, I ran into a problem with the JSON marshaller used by grpc-gateway when used with gogo/protobuf. After some digging , I found a library that works as explained by their description:

“This repo contains a JSON marshaler suitable for use with the gRPC-Gateway when using gogo/protobuf types.”

So to prevent this happening to others as well, this should be downloaded as well:

go get github.com/gogo/gateway

At this point we are ready to generate the gateway Go code. Basically what this means is to generate some code using the protoc plugin that would make life much easier for us. As my previous exercises, and to keep things simple and consistent, I selected ImageService . So I copied image.proto file . You can find it within the containerd installation or directly from here.

There are 2 ways we can tell grpc-gateway how to map REST to gRPC:

We can annotate the proto file directly. This is useful in cases where we own the proto file, so we can update at will
We can create a second file in YAML format that contains this imformation. This is useful in case where we do not own the proto file, hence we avoid messing with it.

Since containerd proto files come from containerd project, I decided to go with the second approach.

The method I wanted to setup REST mapping was Images.List. Based on the images.proto file content, specifically these lines,

syntax = “proto3”;package containerd.services.images.v1;
...
service Images {
….
 rpc List(ListImagesRequest) returns (ListImagesResponse);

I created imagesService.yaml which contains one rule for Images.List method. As can be seen below, selector is the fully qualified name containing package name, service name and method name. Of course in real life, we would have as many as we have methods in our service definition.

type: google.api.Service
config_version: 3http:
  rules:
  - selector: containerd.services.images.v1.Images.List
    get: /v1/images/list

Now that we have a service definition, we run the protoc grpc-gateway plugin with this command:

protoc -I/usr/local/include -I. -I/home/centos/go/src -I/home/centos/go/src/github.com/gogo/protobuf -I/home/centos/go/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis  --grpc-gateway_out=logtostderr=true,grpc_api_configuration=imagesService.yaml:. images.proto

This command created grpc-gateway Go file which contains code that we can use to create a reverse proxy to containerd gRPC server : github.com/containerd/containerd/api/services/images/v1/images.pb.gw.go. This file is also in my github repo along with others.

The important lines in this file are :

package images
….
func RegisterImagesHandlerFromEndpoint(ctx context.Context, mux *runtime.ServeMux, endpoint string, opts []grpc.DialOption) (err error) {

This is the method used to register the images handler with the grpc endpoint. We’ll see clearly how this generated method is used below.

Creating a REST to gRPC Reverse Proxy

So it’s time to code our reverse proxy. I created a file called containerdRestServer.go which can be seen here.

The main portions of the code is explained below:

const (
        restPort   = 8080
        )

Here we define constant for the port REST server will listen to. 8080 is a logical choice for HTTP, we’ll come back to this when configuring TLS.

In main function, we will basically start by pointing to the Unix socket containerd gRPC is listening on with grpc.WithInsecure() option . Since containerd is not listening on a TCP socket, only a UNIX domain socket, as long as we protect this socket file with proper permissions, this is safe.

func main() {
 containerdAddr = “unix:///run/containerd/containerd.sock”
ctx :context.Background()
dopts := []grpc.DialOption{grpc.WithInsecure()}

Then we define a HTTP Request Multiplexer and call the ImagesHandler with that and other parameters:

gwmux := runtime.NewServeMux(runtime.WithMarshalerOption(runtime.MIMEWildcard, m))
        err := images.RegisterImagesHandlerFromEndpoint(ctx, gwmux, containerdAddr, dopts)
mux := http.NewServeMux()
mux.Handle("/", gwmux)
        srv := &http.Server{
                Addr:    restAddr,
                Handler: grpcHandlerFunc(mux),
        }

So we’re basically telling our reverse proxy server what URI patterns it should handle and send to what gRPC backend. The rest of the code is pretty standard. One thing to note here is that the http server I created here is simple HTTP, not encrypted(yet) and no client authentication(yet). We will get to those later. Let’s see this working first. Time to run:

$  go run containerdRestServer.go
Starting HTTP/REST server on port: 8080

Nice, now that it is running, let’s try to use it. For this we can rely on the always very helpful cURL. We need to pass the containerd namespace where we have our images in “containerd-namespace” header. grpc-gateway passes any header that starts with “Grpc-Metadata-” untouched, so we send “Grpc-Metadata-containerd-namespace: examplectr” with cURL and of course we call the URL we defined in our imagesService.yaml file “/v1/images/list”. Here’s the result:

curl --header "Grpc-Metadata-containerd-namespace: examplectr"  http://localhost:8080/v1/images/list{"images":[{"name":"docker.io/envoyproxy/envoy:latest","target":{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:bf7970f469c3d2cd54a472536342bd50df0ddf099ebd51024b7f13016c4ee3c4","size":"2194"},"createdAt":"2019-06-18T20:51:29.107421090Z","updatedAt":"2019-06-18T20:51:29.107421090Z"},{"name":"docker.io/library/alpine:3.6","target":{"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","digest":"sha256:66790a2b79e1ea3e1dabac43990c54aca5d1ddf268d9a5a0285e4167c8b24475","size":"1412"},"createdAt":"2019-06-06T16:14:51.781870201Z","updatedAt":"2019-06-06T16:14:51.781870201Z"},{"name":"docker.io/library/alpine:latest","target":{"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","digest":"sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6","size":"1638"},"createdAt":"2019-06-01T02:08:12.815595360Z","updatedAt":"2019-06-01T02:08:12.815595360Z"},{"name":"docker.io/library/httpd:alpine","target":{"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","digest":"sha256:8a3c608e3e87ab4b818374a5414e63b63097c248693b402dac3616f2ce6d487b","size":"1645"},"createdAt":"2019-06-06T16:16:43.498249217Z","updatedAt":"2019-06-06T16:16:43.498249217Z"}]}

Yay! Now we have a working REST to gRPC reverse proxy that we can talk REST to our containerd. We can use any HTTP client, any code in any language that talks REST without gRPC stub as long as we send proper namespace header and the URI

Securing the environment

What we need next is to close the loop on security. This REST server is wide open to any calls at this time (which is a bad thing), and does not use TLS. So let’s get that in order. There are two things we need to do more:

Configure TLS on the http listener, so communication between clients and our service will be encrypted. For this I’m going to use a certificate from “Let’s Encrypt” free Certificate Authority. I could get away with a self signed certificate here, but the reason is explained in item 2
I’d like to limit to call to this REST-gRPC service to authorized users only. For this I decided to use an API gateway. API gateways are a great way to manage API services by distributing API keys to clients and managing various usage metrics easily. For this one, I decided to use a popular API Gateway from AWS. The reason I could not get away with a self signed certificate above is due to the reason that it needs to be a Trusted CA by AWS API Gateway, and while going through the list of CAs, I realized “Let’s Encrypt” is supported. But essentially , any CA trusted by AWS API Gateway would work.

First we get Let’s Encrypt certificate. I’m not going to go into details of that. It’s explained clearly here : https://letsencrypt.org/getting-started/. Since my OS is CentOS, I used certbot and followed instructions here : https://certbot.eff.org/lets-encrypt/centosrhel7-other

After getting the private key and certificate. I copied them under src/certs/directory as letscert.pem letskey.pem (Obviously not in my GitHub)

I also made a copy of reverse proxy code as containerdRestServerTLS.go to keep working on without changing the other.

Here are main updates:

serverCert, err := tls.LoadX509KeyPair("certs/letscert.pem", "certs/letskey.pem")tlsConfig := tls.Config{
                Certificates: []tls.Certificate{serverCert},
                NextProtos:   []string{"h2"},
        }srv := &http.Server{
                Addr:      restAddr,
                Handler:   grpcHandlerFunc(mux),
                TLSConfig: &tlsConfig,
        }

Let’s also change our port number to 8443, to be a good citizen and obey port conventions.Also change the ListenAnd Serve. So it looks like this:

restPort =8080
...
srv.ListenAndServe()

restPort = 8443
...
 conn, err := net.Listen("tcp", fmt.Sprintf(":%d", restPort))
        if err != nil {
                panic(err)
        }if err = srv.Serve(tls.NewListener(conn, srv.TLSConfig)); err != nil {
     panic(err)
  }

Let’s run this again and try

go run containerdRestServerTLS.go
curl --header "Grpc-Metadata-containerd-namespace: examplectr"  https://YourHostNameThatMatchesCertificateCN:8443/v1/images/list{"images":[{"name":"docker.io/envoyproxy/envoy:latest","target":{"mediaType":"application/vnd.docker.distribution.manifest.v2+json",......

Good! Now we have TLS working.

Setup AWS API Gateway to talk to containerd using gRPC gateway

We can manually define this API in AWS console, but even better is to generate a Swagger OpenAPI file and import into API gateway. Fortunately grpc-gateway comes with an option to generate the swagger

protoc -I/usr/local/include -I. -I/home/centos/go/src -I/home/centos/go/src/github.com/gogo/protobuf -I/home/centos/go/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis  --swagger_out=logtostderr=true,grpc_api_configuration=imagesService.yaml:. images.proto

This swagger protoc plugin generates a swagger.json file which describes our API, in this case images.swagger.json

Next we open AWS API gateway console, and click on “Create API” button. And select the options REST and Import from swagger as below:

Then click on “Import” . After this we see the API in the console. I also renamed this API to “containerd images API”, which kind of makes sense. After we configure GET method:

Let’s deploy this API. Select the API and from the Actions dropdown, select Deploy API

After it is deployed, we can check the “Stages” menu, click on the “test” we deployed. On the right hand it shows the “Invoke URL” which is basically an AWS API Gateway Endpoint. Let’s use cURL to send a request to this URL, adding ‘v1/images/list’ to the end:

curl --header "Grpc-Metadata-containerd-namespace: examplectr" https://SomeEndPoint.execute-api.us-east-1.amazonaws.com/test/v1/images/list
{"images":[{"name":"docker.io/envoyproxy/envoy:latest","target":{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:bf7970f469c3d2cd54a472536342bd50df0ddf099ebd51024b7f13016c4ee3c4","size":"2194"}
….

Sweet! Now we have can have any HTTP client go through API Gateway to talk to containerd.

Limit the service to only authorized users

Next step is to make sure, only API gateway can talk to our REST server, plus generate API key from API gateway and distribute to our REST clients to use to we can control who can use our containerd service. For this we need TLS Client Authentication between API gateway and REST server first.

Let’s go into API Gateway “Client Certificates” page first and click on “Generate Client Certificate.This generates a client certificate with an ID and we can click on “copy” link to copy the content of the certificate to clipboard.

Then create a file apigateway_client_cert.pem under certs directory. Now we have 3 files in there:

ls
apigateway_client_cert.pem letscert.pem letskey.pem

Add these lines to our reverse proxy:

clientCertPool := x509.NewCertPool()
        clientCertPem, err := ioutil.ReadFile("certs/apigateway_client_cert.pem")
        if err != nil {
                panic(err)
        }
        if !clientCertPool.AppendCertsFromPEM(clientCertPem) {
                panic("Can't parse client certificate authority")
        }

Then we need to change our TLS config to authorize TLS clients:

tlsConfig := tls.Config{
                Certificates: []tls.Certificate{cert},
                NextProtos:   []string{"h2"},
                ClientAuth:   tls.RequireAndVerifyClientCert,
                ClientCAs:    clientCertPool,
        }

After this, run the REST proxy server again.Try with curl gives an error as expected:

curl --header "Grpc-Metadata-containerd-namespace: examplectr" https://SomeEndPoint.execute-api.us-east-1.amazonaws.com/test/v1/images/list
{“message”: “Internal server error”}

Because, API Gateway is not sending client cert yet. So we need to tell API Gateway to use the client certificate we generated. Click on the “Stages” then “test” again. At the bottom of the page we see the Client Certificate drop down, so we select the one we just created and “Save Changes”.

Try again:

curl --header "Grpc-Metadata-containerd-namespace: examplectr" https://SomeEndPoint.execute-api.us-east-1.amazonaws.com/test/v1/images/list
{"images":[{"name":"docker.io/envoyproxy/envoy:latest","
….

Nice!

One last step is left. Let’s click on “API Keys” on API Gateway console, select “Create API Key” from button drop down

After saving, let’s get the API key generated and put it in a safe place. Click on “Show” to view

Let’s go back to our API and click on “List” resource. Currently it shows no API key needed:

Click on “Get”

And then on “Method Request”

We change “API Key Required” to true and click on check mark to save.

Then we deploy API one more time. I’ve experienced it did not take effect right away, so let’s give it a few minutes.

curl --header "Grpc-Metadata-containerd-namespace: examplectr" https://SomeEndPoint.execute-api.us-east-1.amazonaws.com/test/v1/images/list
{“message”:”Forbidden”}

Alright, now let’s create “API Usage plan” , select that menu form the left

Let’s not deal with throttling or quotas at this point. So we uncheck them. Then we add a “Stage”

Then we add the API Key

As (hopefully) our last curl command, we run with our API Key we generated. To do this we send “x-api-key” header with the value set to the API key

curl --header "Grpc-Metadata-containerd-namespace: examplectr" --header "x-api-key :API Key Here" https://SomeEndPoint.execute-api.us-east-1.amazonaws.com/test/v1/images/list
{"images":[{"name":"docker.io/envoyproxy/envoy:latest","target":{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","
….

Yay again!

So at this point, here is how our architecture looks like:

We have a REST to gRPC reverse proxy running using grpc-gateway project and it is communicating with containerd services using gRPC interface. It is configured with TLS mutual authentication with regards to communicating with HTTP clients We also have an AWS API Gateway API configured with client certificate, and usage plan for containerd customers. We generated an API key that was distributed to the customer(cURL) which was able to send a HTTP1.1/REST request using this key and get a proper response back

Happy computing!