How to Deploy Envoy as a Sidecar Proxy on Kubernetes

It only takes 3 steps!

6 min readDec 17, 2022

In my previous post, we saw how to deploy Envoy as an edge proxy. Now we will extend the same example to deploy Envoy as a sidecar proxy.

The complete code is available on my Github repo

3 Steps to Introduce a Sidecar Proxy

Create an Envoy.yaml that listens to traffic and routes it to the service.
Containerize Envoy.
Create the K8s yaml, which includes:
→ K8s deployment with both the Envoy and Service containers
→ K8s service that exposes only the Envoy container (so that the service can be reached only via Envoy)

We do these 3 steps below for the existing service, and we end up with:

The Flask app has both an edge and sidecar proxy!

Step 1

Envoy.yaml:

static_resources:
  listeners:
  # ingress
  - name: mainapp_sidecar_listener
    address:
      socket_address:
        # Entrypoint for service through Envoy
        address: 0.0.0.0
        port_value: 5199
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          # used when emitting stats
          stat_prefix: mainapp_sidecar_hcm_filter
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router 
          route_config:
            name: mainapp_sidecar_http_route_config
            virtual_hosts:
            # name used when emitting stats, not imp for routing itself
            - name: mainapp_sidecar_virtual_host
              domains: 
              - "*"
              routes:
              - name:
                match:
                  prefix: "/"
                route:
                  cluster: mainapp_service
  clusters:
  - name: mainapp_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: mainapp_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                # reroute to service container in the same K8s deployment
                address: 127.0.0.1
                port_value: 5200

Step 2

Dockerfile:

FROM envoyproxy/envoy:v1.24.0

# override the existing default envoy.yaml 
COPY envoy.yaml etc/envoy/envoy.yaml

RUN chmod go+r /etc/envoy/envoy.yaml

CMD ["/usr/local/bin/envoy", "-c", "/etc/envoy/envoy.yaml"]

# Let's build the Envoy sidecar in microsvc/mainapp-enovy-sidecar
$ docker build -f Dockerfile -t mainapp-envoy-sidecar .

# Let's also build the Flask app in microsvc/mainapp-flask
$ docker build -f docker/Dockerfile -t livescores-microsvc .

Step 3

K8s yaml (deployment-svc.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mainapp-envoy-sidecar-deployment
  labels:
    app: mainapp-envoy-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mainapp-envoy-sidecar
  template:
    metadata:
      labels:
        app: mainapp-envoy-sidecar
    # The imp stuff
    spec:
      # Deployment has 2 containers!
      containers:
      - name: mainapp-envoy-sidecar
        image: mainapp-envoy-sidecar:latest
        imagePullPolicy: Never
        ports:
        - containerPort: 5199
          name: envoy-port
        
      - name: livescores-microsvc
        image: livescores-microsvc:latest
        imagePullPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
  name: mainapp-envoy-sidecar-svc
spec:
  type: ClusterIP
  selector:
    app: mainapp-envoy-sidecar
  ports:
  # Service only exposes the Envoy container
  - name: mainapp-envoy-sidecar-svc-port
    protocol: TCP
    port: 5199
    targetPort: envoy-port

$ minkube start
$ minikube image load mainapp-envoy-sidecar
$ minikube image load livescores-microsvc

# K8s yaml file is in microsvc/mainapp-flask
$ kubectl apply -f kubernetes/deployment-svc.yaml

# Need to run this to expose the service - input password too
$ minikube tunnel

# Now access on the browser at http://localhost

Make it More Interesting

We can break away the football scores into a separate (micro)service that returns a JSON response which we then format and present as before. For this, I made a copy of the Flask app, moved the football scores logic into it, and used Flask's jsonify() to return a JSON response.

The code is in the football-flask directory.

Now, we repeat the same 3 steps as above to introduce Envoy as a sidecar proxy so that we end up with 2 sidecar proxies for the 2 services:

NOTE: Now we have to handle the outgoing call from the Flask app via the sidecar proxy too! So, the Envoy yaml of the Flask app needs to be modified and redeployed. [But I haven’t done so to keep it simple]

Step 1

Envoy.yaml

static_resources:
  listeners:
  # ingress
  - name: football_sidecar_listener
    address:
      socket_address:
        # Entrypoint for service through Envoy
        address: 0.0.0.0
        port_value: 6199
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          # used when emitting stats
          stat_prefix: football_sidecar_hcm_filter
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: football_sidecar_http_route_config
            virtual_hosts:
            # name used when emitting stats, not imp for routing itself
            - name: football_sidecar_virtual_host
              domains: 
              - "*"
              routes:
              - name:
                match:
                  prefix: "/"
                route:
                  cluster: football_service
  clusters:
  - name: football_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: football_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                # reroute to service container in the same K8s deployment
                address: 127.0.0.1
                port_value: 6200

Step 2

Dockerfile:

FROM envoyproxy/envoy:v1.24.0

# override the existing default envoy.yaml 
COPY envoy.yaml etc/envoy/envoy.yaml

RUN chmod go+r /etc/envoy/envoy.yaml

CMD ["/usr/local/bin/envoy", "-c", "/etc/envoy/envoy.yaml"]

# Let's build the Envoy sidecar in microsvc/football-enovy-sidecar
$ docker build -f Dockerfile -t football-envoy-sidecar .

# Let's also build the Flask app in microsvc/football-flask
$ docker build -f Dockerfile -t footballscores .

Step 3

K8s yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: football-envoy-sidecar-deployment
  labels:
    app: football-envoy-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: football-envoy-sidecar
  template:
    metadata:
      labels:
        app: football-envoy-sidecar
    # The imp stuff
    spec:
      # Deployment has 2 containers!
      containers:
      - name: football-envoy-sidecar
        image: football-envoy-sidecar:latest
        imagePullPolicy: Never
        ports:
        - containerPort: 6199
          name: envoy-port

      - name: footballscores
        image: footballscores:latest
        imagePullPolicy: Never

---
apiVersion: v1
kind: Service
metadata:
  name: football-envoy-sidecar-svc
spec:
  type: ClusterIP
  selector:
    app: football-envoy-sidecar
  ports:
  # Service exposes only the Envoy container
  - name: football-envoy-sidecar-svc-port
    protocol: TCP
    port: 6199
    targetPort: envoy-port

$ minikube image load football-envoy-sidecar
$ minikube image load footballscores

# K8s yaml file is in microsvc/football-flask
$ kubectl apply -f deployment-svc.yaml

# Need to run this to expose the service - give password
$ minikube tunnel

# Now access on the browser at http://localhost

Make it Even More Interesting

We can repeat the above steps to separate the cricket scores into yet another Flask (micro)service. However, the advantage of microservices is that you can use different languages for different services. So, I used Ballerina for this service. It’s a relatively obscure language, but I wanted to see its ability to auto-generate Docker images and K8s yaml files.

The code is in the cricket-ballerina directory.
Autogenerate the Docker image with: bal build --cloud="docker"
But I didn’t like the auto-generated K8s yaml, so I added my own.

Once this is done, we follow the same 3 steps again, and we end up with the target state:

The Envoy code for this part can be found here.

Summary

With 3 steps, you can add Envoy as a sidecar proxy to your services — configure Envoy and containerize it, then put it alongside your service with a K8s deployment & expose it via a K8s service.

References:

Flask: Hitting other services with Flask
Ballerina: Basics, examples, auto-generating Docker & K8s artifacts
Kubernetes: Env variables, DNS for services
Sidecar: Excellent post on Envoy as a sidecar in containers, deploying 2 containers in a single K8s deployment

How to Deploy Envoy as a Sidecar Proxy on Kubernetes

It only takes 3 steps!

3 Steps to Introduce a Sidecar Proxy

Step 1

Step 2

Step 3

Make it More Interesting

Step 1

Step 2

Step 3

Make it Even More Interesting

Summary

References:

Written by Viggnah Selvaraj