How to Deploy Envoy as an Edge Proxy on Kubernetes

An easy 6-step practical guide

Photo by John Gibbons on Unsplash

Envoy will be introduced in step 4, as shown below. Let’s get started!

Overview of the 6 steps

Step 1

Prerequisite: flask

You must have (or create) a service or app. I have created a Flask app that serves some static data. You can get it along with the rest of the code at my Github repo.

• Install the required packages and run the app locally with:

$ cd envoy-examples
$ python3 -m venv venv
$ . venv/bin/activate
$ pip3 install Flask

# if `flask` command does not work, use `python3 -m flask`
$ flask --app sportshome.py --debug run --host=0.0.0.0 --port=5200

The Flask app running locally

Step 2

Prerequisite: docker

Let’s put the Flask app into a Docker container and run it to ensure it’s working as expected.

• First, install Docker Desktop or the Docker engine.
• Then we will use the following Dockerfile:

FROM python:3.8.9-slim

WORKDIR /livescores

COPY requirements.txt requirements.txt

RUN pip3 install -r requirements.txt

COPY . /livescores

EXPOSE 5200

CMD [ "flask", "--app", "sportshome.py", "--debug", "run", "--host=0.0.0.0", "--port=5200"]

• Go into the flask-app folder, and build and run it:

$ docker build -f docker/Dockerfile -t livescores .
$ docker run --rm -p 5200:5200 livescores

The Flask app running inside a Docker container is accessible via the browser

Step 3

Prerequisite: minikube

Let’s run the container on a Kubernetes(K8s) cluster. First, install minikube to create a local K8s cluster. Optionally, install kubectl or use minikube’s kubectl instead.

• Now, start the cluster and load the image. Loading the image allows us to use the local Docker image:

$ minkube start

$ minikube image load livescores
# Verify the image has loaded
$ minikube image ls

• Create a K8s deployment and service for the Flask app using the deployment-svc.yaml inside the flask-app/kubernetes folder.
• Note that this deployment has 2 replicas, and the service has to be type: LoadBalancer. There is also an environment variable called POD_IP that will let us see which of the 2 replicas we are actually hitting.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: livescores-deployment
  labels:
    app: livescores
spec:
  replicas: 2
  selector:
    matchLabels:
      app: livescores
  template:
    metadata:
      labels:
        app: livescores
    # The imp stuff
    spec:
      containers:
      - name: livescores
        image: livescores:latest
        ports:
        - containerPort: 5200
          name: flask-port
        imagePullPolicy: Never
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: livescores-service
spec:
  type: LoadBalancer
  selector:
    app: livescores
  ports:
  - name: livescores-svc-port
    protocol: TCP
    port: 4200
    targetPort: flask-port

# Need to run this to expose the service
$ minikube tunnel

# Alternative: minikube kubectl -- apply -f deployment-svc.yaml
$ kubectl apply -f deployment-svc.yaml

# Verify that the deployment and service(svc) have been created
$ kubectl get deployment
$ kubectl get svc

The Flask app running on a minikube cluster is accessible via the browser

Step 4

Prerequisite: envoy

You can skip this step if you don’t want to try out Envoy locally first.

Install Envoy.

• Let’s run the Flask app as shown in step 1 again.
• Then, we have to run Envoy with a yaml config file:

static_resources:
  listeners:
  - name: lss_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 3200
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          # used when emitting stats
          stat_prefix: lss_hcm_filter
          
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: lss_http_route_config
            virtual_hosts:
            # name used when emitting stats, not imp for routing itself
            - name: lss_virtual_host
              domains: 
              - "*"
              routes:
              - name:
                match:
                  prefix: "/"
                route:
                  cluster: lss_service
  clusters:
  - name: lss_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: lss_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 5200

# Inside the envoy-r-proxy directory
$ enovy -c envoy.yaml

Using Envoy as a reverse proxy on the edge to access the Flask app

Step 5

Let’s containerize Envoy so it can be deployed on the minikube cluster.

• Change the cluster’s endpoint in the envoy.yaml file since the service we are going to hit is now a K8s service, not a local deployment like in step 4.

clusters:
  - name: lss_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: lss_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: livescores-service
                port_value: 4200

• The Dockerfile for the Envoy proxy:

FROM envoyproxy/envoy:v1.24.0

COPY envoy.yaml etc/envoy/envoy.yaml

RUN chmod go+r /etc/envoy/envoy.yaml

CMD ["/usr/local/bin/envoy", "-c", "/etc/envoy/envoy.yaml"]

• Go into the envoy-r-proxy folder, and build the image:

$ docker build -f Dockerfile -t envoyrproxy .

Step 6

Let’s complete the setup by deploying Envoy on the minikube Kubernetes cluster.

• Load the image to minikube:

$ minikube image load envoyrproxy

• Create a deployment and service for the Envoy proxy using the deployment-svc.yaml file inside the envoy-r-proxy directory:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoyrproxy-deployment
  labels:
    app: envoyrproxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: envoyrproxy
  template:
    metadata:
      labels:
        app: envoyrproxy
    # The imp stuff
    spec:
      containers:
      - name: envoyrproxy
        image: envoyrproxy:latest
        ports:
        - containerPort: 3200
          name: envoy-port
        imagePullPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
  name: envoyrproxy-service
spec:
  type: LoadBalancer
  selector:
    app: envoyrproxy
  ports:
  - name: envoyrproxy-svc-port
    protocol: TCP
    port: 80
    targetPort: envoy-port

$ kubectl apply -f deployment-svc.yaml

• Remember to keep the minikube tunnel process running. Switch to that shell and give your machine password since we are listening on port 80 now. You may need to stop and rerun as sudo minikube tunnel if it is not working the first time.
• Now, we can access the Flask app via the browser on http://localhost. But the Flask app is also accessible directly via http://localhost:4200!
• Let’s stop direct access by changing the Flask app K8s service yaml in step 3. Change type: LoadBalancer to type: ClusterIP like below and run kubectl again:

---
apiVersion: v1
kind: Service
metadata:
  name: livescores-service
spec:
  type: ClusterIP
  selector:
    app: livescores
  ports:
  - name: livescores-svc-port
    protocol: TCP
    port: 4200
    targetPort: flask-port

$ kubectl apply -f deployment-svc.yaml

• Now, the Flask app is only accessible via the Envoy proxy.

The Envoy proxy acts as a reverse proxy on the edge. The Flask app is only accessible via Envoy!

TL;DR — The Flask app and Envoy are containerized and deployed on a local K8s cluster (minikube). The Flask app is accessible on the browser only via the Envoy proxy. Follow all 6 steps to get a sample Flask app proxied via Envoy on the edge. Follow from step 4 onwards for the steps involving Envoy.

You can read the detailed Envoy docs or my previous post with some useful references to learn more about Envoy proxy.

References:

• Flask quickstart guide
• Dockerizing a Flask app — reference1, reference2
• Using local Docker images with minikube
• Kubernetes deployments & services
• Accessing apps on minikube via minikube tunnel
• Static configuration of Envoy & debugging Envoy