File Based Dynamic Configuration of Routes in Envoy Proxy

Published in

Compendium

3 min readMar 12, 2019

As developers are increasingly involved in the configuration of edge proxies, like Envoy, it is important that these appliances provide a developer experience that supports the modern developers’ workflow. With the dynamic service discovery capabilities, Envoy makes this possible. All aspects of the proxy except a static bootstrap configuration can be dynamically configured. This means that the configuration of an Envoy proxy can be updated without cold or hot restart. This is in many cases the preferred and easiest method of handling frequent configuration changes.

The configuration can either be provided by a configuration server, or by files. There are some open source and commercial implementations of such servers. If desired, projects can implement their own, based on available libraries, implementations and examples (envoyproxy/go-control-plane, envoyproxy/java-control-plane). If the requirements do not necessitate a server based solution, a simple file based approach might be sufficient. A file with the new configuration is created, and in a file system move operation the current file is overwritten with the new configuration. Using inotify to monitor the file for changes, Envoy will load the new configuration, and apply it if it is valid. If it is not valid, the previous state will remain active, and a corresponding message will be written to the log.

The whole family of discovery services is designated as xDS. This post shows how the route discovery service (RDS) can be implemented using files as configuration source instead of servers.

The code accompanying this post is available on GitHub. It is based on the front proxy example code from the Envoy proxy repository, documented in the
envoy docs.

In the Envoy configuration file, find the field route_config the envoy.http_connection_manager filter that you would like to replace by a dynamic configuration:

- name: envoy.http_connection_manager
  config:
    codec_type: auto
    stat_prefix: ingress_http
    route_config:
      name: local_route
      virtual_hosts:
      - name: backend
        domains:
        - "*"
        routes:
        - match:
            prefix: "/service/1"
          route:
            cluster: service1
        - match:
            prefix: "/service/2"
          route:
            cluster: service2
    http_filters:
    - name: envoy.router
      config: {}

Replace this field with a field called rds that points to the file containing the corresponding configuration:

- name: envoy.http_connection_manager
  config:
    codec_type: auto
    stat_prefix: ingress_http
    rds:
      route_config_name: local_route
      config_source:
        path: /etc/rds/rds.yaml
    http_filters:
    - name: envoy.router
      config: {}

The contents of the file rds.yaml is

version_info: "0"
resources:
- "@type": type.googleapis.com/envoy.api.v2.RouteConfiguration
  name: local_route
  virtual_hosts:
  - name: backend
    domains:
    - "*"
    routes:
    - match:
        prefix: "/service/1"
      route:
        cluster: service1
    - match:
        prefix: "/service/2"
      route:
        cluster: service2

Now, if the configuration needs to be changed, a file with a new configuration can be created, say rds-new.yaml, and then moved over the existing file:

mv rds-new.yaml rds.yaml

Envoy will pick up the change, validate and activate the new configuration.

Further reading and resources:

Originally published at medium.com.

File Based Dynamic Configuration of Routes in Envoy Proxy

Written by Michael Gfeller