Secure Browser Communication with Containerd Using gRPC, Envoy and OAuth 2.0
In my previous post, I explained how we can have browsers communicate with containerd over gRPC protocol.
That was an initial setup showing it is possible to have a browser client to containerd using Envoy proxy in the middle. In this post, I will explore how we can improve this setup by setting up TLS between Envoy and our browser so we can encrypt the data and then implement an authentication/authorization system to secure access to containerd.
Configure Envoy to use HTTPS with a self-signed certificate created with OpenSSL
First step is to create a private key and certificate (that identifies our server). We will not go through the effort of getting our certificate by a Trusted Certificate Authority here, but that’s what you would want to do in Production. Here I’ll create a self-signed certificate, which will generate a warning in the browser, but that’s OK for testing purpose. I’ll create these two files under the same location where I kept the envoy.yaml file to keep things simple. Usually these files go under sundirectories of /etc/pki. Most important thing is to keep the private key very secure.
# openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out /etc/envoy/EnvoyCert.crt -keyout /etc/envoy/Envoy.key
Generating a 4096 bit RSA private key
………………..++
…………++
writing new private key to ‘/etc/pki/tls/private/Envoy.key’
— — -
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
— — -
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:NY
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:Mark IO
Organizational Unit Name (eg, section) []:Tech
Common Name (eg, your name or your server’s hostname) []:envoy.mark.io
Email Address []:ssl@mark.io
# pwd
/etc/envoy
# ls -l
total 12
-rw-r--r--. 1 root root 2098 Jun 24 20:48 EnvoyCert.crt
-rw-r--r--. 1 root root 3272 Jun 24 20:48 Envoy.key
-rw-r--r--. 1 root root 2101 Jun 24 20:43 envoy.yamlAt this time we need to configure Envoy configuration file envoy.yaml for TLS. I copied the full file here, updates are in BOLD. I changed the port from 8080 to 443 since it is conventionally the port used with SSL/TLS.
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }static_resources:
listeners:
— name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
— filters:
— name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
— name: local_service
domains: [“*”]
routes:
— match: { prefix: “/containerd.services” }
route:
cluster: containerd_service
max_grpc_timeout: 0s
— match: { prefix: “/” }
route:
cluster: default_service
cors:
allow_origin:
— “*”
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: containerd-namespace,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: “1728000”
expose_headers: containerd-namespace,custom-header-1,grpc-status,grpc-message
enabled: true
http_filters:
— name: envoy.grpc_web
— name: envoy.cors
— name: envoy.router
tls_context:
common_tls_context:
tls_certificates:
— certificate_chain: {“filename”: “/etc/envoy/EnvoyCert.crt”}
private_key: {“filename”: “/etc/envoy/Envoy.key”}
clusters:
— name: containerd_service
connect_timeout: 0.25s
type: static
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{“pipe”: {“path”: “/run/containerd/containerd.sock”}}]
— name: default_service
connect_timeout: 0.25s
type: static
lb_policy: round_robin
hosts: [{ socket_address: { address: 127.0.0.1, port_value: 8081 }}]
Now, let’s restart our Envoy proxy again:
sudo /usr/local/bin/ctr -namespace examplectr task start myenvoyThis is assuming task was now running but the container exists. If the container does not exist, we need to create the container again using the commands from the previous exercise:
sudo /usr/local/bin/ctr -namespace examplectr image pull docker.io/envoyproxy/envoy:latest
sudo /usr/local/bin/ctr -namespace examplectr container create — net-host — mount type=bind,src=/run/containerd,dst=/run/containerd,options=rbind:ro — mount type=bind,src=/etc/envoy,dst=/etc/envoy,options=rbind:ro docker.io/envoyproxy/envoy:latest myenvoyUpdate client script and test
In previous post we created a client.js file that sends a gRPC-Web request to list images in containerd. In the script we used http for sending request to Envoy. Since we now converted Envoy to use HTTPS, we need to update that line to https. I will omit :443 since that is the default port so we do not need to specify that:
var client = new ImagesClient(‘https:{Our Server Public IP}’);Since we updated this file, let’s package it up again using npx:
npx webpack client.jsAt this time we are ready. Let’s try to access this using the browser:
https://{Our Server Public IP}/First time we access this we get a warning as expected, but nothing to worry about.
Click “Advanced” button and “Proceed to …” link
This brings us to familiar screen from last exercise, except this time it is HTTPS!
Use GitHub user accounts to secure sign in to containerd web interface with OAuth 2.0
Instead of building a whole new user authentication system from scratch, I will utilize GitHub’s OAuth system to have users sign in with their GitHub username and password.
To start, let’s create a new OAuth application within GitHub developer settings
After this we are provided a CLIENT_ID and CLIENT_SECRET to use later in our code
Build Envoy External Authorization Server
Envoy supports external authorization filter which is explained in Envoy documentation:
The External authorization filter calls an authorization service to check if the incoming request is authorized or not. The filter can be either configured as a network filter, or as a HTTP filter or both. If the request is deemed unauthorized by the network filter then the connection will be closed. If the request is deemed unauthorized at the HTTP filter the request will be denied with 403 (Forbidden) response.
Envoy does this by making a gRPC call to an authorization service, passes request and some more data and receives a response that determines whether access to upstream is allowed or not and also add or replace HTTP headers and body.
So let’s create an authorization service that Envoy can call using gRPC protocol. Envoy provides proto files here. We can use any language that is within gRPC world, and I will use Go in this exercise. To start, we need to get the libraries we will use. We need two repos from Envoy and ‘sessions’ package of Gorilla Web Toolkit (to implement user sessions as we do not want user to login with each request)
go get github.com/envoyproxy/data-plane-api
go get github.com/envoyproxy/protoc-gen-validate
go get github.com/gorilla/sessionsWe also need the referenced Google RPC status library. I did not want to clone the whole repo (it’s big), so I just created a directory for status.proto and downloaded the file there
cd $GOPATH/github.com (GOPATH is where you keep Go packages something like /home/centos/go)mkdir -p googleapis/google/rpc
wget -O googleapis/google/rpc/status.proto https://raw.githubusercontent.com/googleapis/googleapis/master/google/rpc/status.proto
Create a directory to contain our code
$mkdir -p envoy_auth_server/src
$ cd envoy_auth_serverAfter this let’s generate the Go gRPC and Protobuf code from these proto files (I’m running this in 4 distinct commands, as I had a problem running all 4 at the same time. There probably a better way to do this)
$export PROTO_ROOT=/home/centos/go
$ protoc --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/protoc-gen-validate --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/data-plane-api --proto_path=$PROTO_ROOT/src/github.com/googleapis --go_out=plugins=grpc:src $PROTO_ROOT/src/github.com/envoyproxy/data-plane-api/envoy/service/auth/v2/*.proto
protoc --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/protoc-gen-validate --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/data-plane-api --proto_path=$PROTO_ROOT/src/github.com/googleapis --go_out=plugins=grpc:src $PROTO_ROOT/src/github.com/envoyproxy/data-plane-api/envoy/api/v2/core/*.proto
protoc --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/protoc-gen-validate --proto_path=$PROTO_ROOT/src/github.com/envoyproxy/data-plane-api --proto_path=$PROTO_ROOT/src/github.com/googleapis --go_out=plugins=grpc:src $PROTO_ROOT/src/github.com/envoyproxy/data-plane-api/envoy/type/*.protoWe need to do a little trick here for gogoproto, since the way it's imported in github.com/envoyproxy/data-plane-api/envoy/type/percent.proto is "gogoproto/gogo.proto", without fully qualified name:
protoc --proto_path=/tmp/containerd_protos/gogoproto --go_out=./src gogo.proto
mv src/github.com/gogo/protobuf/gogoproto src/
Now, create a file called ‘envoy-auth-server.go’ and copy content from where I uploaded to GitHub. Then set GOPATH and build
export GOPATH=/home/centos/go:`pwd`
go build src/envoy-auth-server.go
./envoy-auth-server
2019/07/03 22:32:03 Starting envoy Auth gRPC ServiceGreat! Now we have a running Envoy Authorization Server listening on port 10003
Configure Envoy to Use Our Authorization Server
At this point, we can tell Envoy to start using this server. We need to update our envoy.yaml file . I copied the file to GitHub, so you can download from here.
The main parts are below. Here we define the envoy.ext_authz HTTP filter and our cluster to serve it:
http_filters:
- name: envoy.ext_authz
config:
grpc_service:
envoy_grpc:
cluster_name: ext-authz
timeout: 0.5s
...
cluster_name: ext-authz
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 10003We also define a URI prefix ‘/public’ to serve content that does not require authorization. We will use this directory to store login page and related content:
- match: { prefix: "/public" }
per_filter_config:
envoy.ext_authz:
disabled: true
route:
cluster: default_serviceLogin Page
This page will be shown when an unauthenticated used requests protected content. Since we have only one OAuth2 provider in this exercise, we will only have GitHub link to let user select GitHub.com for authentication
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>containerd gRPC-Web Example</title>
</head>
<body>
Please login using buttons below <br/><br/>
<a href="https://github.com/login/oauth/authorize?client_id=b01f19813cf4178e4cf8"><img src="GitHub.png"/><br/>GitHub</a>
</body>
</html>
When the user clicks on the link, this page is shown first time to ask user to allow our app to access GitHub credential
Once user allows by clicking the button, user is taken back to our site, this time with a temporary token to exchange for an Access Token which is done by the setUserFromGitHub function.
After this user is directed to home page and if the user is in the allowed users list , is able to access it. Allowed users list is a simple array within the envoy-auth-server.go. You can change it to our user to test.
var allowedUsers = [1]string{“mark-kose”}
And now, as can be see above, we are able to limit access to containerd by using Envoy Proxy’s External Authorization service using gRPC.
Happy Containerizing!
