Don’t Repeat Yourself with Anchors, Aliases and Extensions in Docker Compose Files

Docker Compose files are a great way to define multiple containers and services that work together as a stack. But, it can be easy to end up with repeated blocks in stacks with lots of similar services or configurations. Don’t Repeat Yourself (DRY) by leveraging YAML aliases and anchors, and the upcoming extension fields feature in the Docker Compose file format.

What’s the Problem?

Docker Compose files list zero or more services that are deployed together. A simple example might consist of a web server, app server, and database for a generic application.

version: '3.3'
services:
  web:
    image: nginx
    port:
      - 80:80
  app:
    image: my-app
    environment:
      - DB_HOST=db
  db:
    image: postgres

Each service in this example is unique and do not share any options. The Docker image for each service is different. As are configuration options like the published port on the web service and the environment variable on the app service. The services are not related to each other and do not have repeated sections between them.

As your stack grows, you may start to find common configurations that span multiple services, especially in microservice architectures. The example Docker Compose file for OpenFaaS—a serverless framework for Docker & Kubernetes—defines eight demo services for functions that can be called via the FaaS gateway. Here is an excerpt from the stack for the nodeinfo and echoit services.

services:
  # Node.js gives OS info about the node (Host)
  nodeinfo:
    image: functions/nodeinfo:latest
    labels:
      function: "true"
    depends_on:
      - gateway
    networks:
      - functions
    environment:
      no_proxy: "gateway"
      https_proxy: $https_proxy
    deploy:
      placement:
        constraints:
          - 'node.platform.os == linux'
  # Uses `cat` to echo back response, fastest function to execute.
  echoit:
    image: functions/alpine:health
    labels:
      function: "true"
    depends_on:
      - gateway
    networks:
      - functions
    environment:
      fprocess: "cat"
      no_proxy: "gateway"
      https_proxy: $https_proxy
    deploy:
      placement:
        constraints:
          - 'node.platform.os == linux'

The labels, depends_on, networks, and deploy options are the same between the two services, and in fact identical across 7 of the 8 services in the stack. Repeating these details can be cumbersome and lead to errors. Changes have to be duplicated many times, and mistakes or omissions can happen by accident. In the context of DRY, repeating yourself is known as WET, or “we enjoy typing” (among other interpretations).

YAML Aliases and Anchors to the Rescue

Docker Compose files are written in YAML. Like JSON, YAML lets you describe typical data types and structures such as strings, numbers, booleans, sequences, and mappings (in fact, YAML is a superset of JSON). But, YAML has many extra tricks that take it beyond just plain old data serialization. One of these goodies is Anchors and Aliases.

Anchors and aliases let you identify an item with an anchor in a YAML document, and then refer to that item with an alias later in the same document. Anchors are identified by an & character, and aliases by an * character. Here is an example showing an item in a list that is identified by an anchor named flag, that is then refered to later on in the list.

- &flag Apple
- Beachball
- Cartoon
- Duckface
- *flag

When this list is read by a YAML parser, the value identified by the anchor (Apple) will be filled at the alias when the YAML is composed. The literal anchor and alias are discarded so they don’t appear in the final result.

- Apple
- Beachball
- Cartoon
- Duckface
- Apple

Anchors can be used repeatedly by multiple aliases. So, the value Apple can be repeated several times in the example list. Anchor names can also be reused. Aliases will refer to the most recent instance of an anchor.

DRY Docker Compose Services with Anchors and Aliases

Earlier on, we saw that the example Docker Compose file for OpenFaaS repeated several details between its services for functions. Let’s use anchors and aliases to cut down the repetition.

services:
  # Node.js gives OS info about the node (Host)
  nodeinfo: &function
    image: functions/nodeinfo:latest
    labels:
      function: "true"
    depends_on:
      - gateway
    networks:
      - functions
    environment:
      no_proxy: "gateway"
      https_proxy: $https_proxy
    deploy:
      placement:
        constraints:
          - 'node.platform.os == linux'
  # Uses `cat` to echo back response, fastest function to execute.
  echoit:
    <<: *function
    image: functions/alpine:health
    environment:
      fprocess: "cat"
      no_proxy: "gateway"
      https_proxy: $https_proxy

The anchor and alias here look a bit different than the simple list example! In this case, we are not referring to a scalar value (“Apple” in the list). Instead, we want to refer to the nodeinfo service mapping. To do that, the &function anchor is placed after nodeinfo is declared, and before the first key-value pair in the mapping.

In echoit, the first key-value pair uses << as the key. This is a special key that indicates key-values from another mapping should be merged into this mapping. That other mapping is comes from the alias *function in this case. So the key-values for image, labels, depends_on, network, environment, and deploy from nodeinfo are merged into echoit, first. Then we overide the values for image and environment because they differ from nodeinfo. The end result is the same definitions for nodeinfo and echoit as earlier, but without repeating items like labels and network.

Using Extension Fields to Create Base Services

In the previous example, the nodeinfo service was used to provide the base definition for echoit, which then overrode specific key-values like image and environment. This strategy can be problematic if the services only share some details, but differ in all other aspects. Now, instead of having to keep track of repeated items, you end up with a potentially messier problem of keeping track of overrides. Worse yet, the anchored service may set an option that you don’t want set at all in the aliased service. There’s no way to skip a key-value while merging in from another mapping.

Docker Compose file format 3.4 introduces support for extension fields. Any top-level key starting with x- in a Docker Compose file will be ignored by Docker Compose and the Docker engine. You can use extensions to declare a partial service containing only the shared options. Because they will be ignored, you can cleanly separate out services meant to be used as anchors only, from services that are actually meant to be deployed.

Here are the same services as before, but with a new x-function service fragment that is declared as an extension. Note that x-function is declared at the top-level, not under services.

# Basic function options
x-function: &function
  labels:
    function: "true"
  depends_on:
    - gateway
  networks:
    - functions
  deploy:
    placement:
      constraints:
        - 'node.platform.os == linux'
services:
  # Node.js gives OS info about the node (Host)
  nodeinfo:
    <<: *function
    image: functions/nodeinfo:latest
    environment:
      no_proxy: "gateway"
      https_proxy: $https_proxy
  # Uses `cat` to echo back response, fastest function to execute.
  echoit:
    <<: *function
    image: functions/alpine:health
    environment:
      fprocess: "cat"
      no_proxy: "gateway"
      https_proxy: $https_proxy

The x-function service isn’t a valid service definition in a Docker Compose file (it’s missing image). But, that doesn’t matter because it’s named starting with x-, which Docker will treat as an extension and ignore. The nodeinfo and echoit services both merge it in, then set their specific image and environment. Changes to the nodeinfo service will not have side effects on echoit, because echoit no longer merges in nodeinfo.

Extensions also make it easy to use multiple anchors and aliases to compose service definitions. For example, the OpenFaaS demo stack might one day include Windows-based services. Instead of setting the placement constraint for Linux in x-function, separate OS placement extensions can be created and used.

# Basic function options
x-function: &function
  labels:
    function: "true"
  depends_on:
    - gateway
  networks:
    - functions
# Linux placement
x-linux: &linux-only
  deploy:
    placement:
      constraints:
        - 'node.platform.os == linux'
# Windows placement
x-windows: &windows-only
  deploy:
    placement:
      constraints:
        - 'node.platform.os == windows'
services:
  # Node.js gives OS info about the node (Host)
  nodeinfo:
    <<: *function
    <<: *linux-only
    image: functions/nodeinfo:latest
    environment:
      no_proxy: "gateway"
      https_proxy: $https_proxy
  # Uses `cat` to echo back response, fastest function to execute.
  echoit:
    <<: *function
    <<: *linux-only
    image: functions/alpine:health
    environment:
      fprocess: "cat"
      no_proxy: "gateway"
      https_proxy: $https_proxy

Now, services like nodeinfo and echoit can easily be set up as FaaS function services for Linux or Windows hosts. Other services can just as easily use the linux-only and windows-only anchors without taking on the values in the function anchor.

Extensions can be used for any part of a Docker Compose file. The concept here for services apply just as equally to networks, volumes, and environments, for example.

Summary

YAML anchors and aliases let you reference and use the same data multiple times within a single YAML document. You can use them to repeat simple scalar values (such as strings) or key-value pairs in mappings. For mappings, the special << key lets you merge in an aliased mapping.

Docker Compose file format 3.4 adds support for extension fields: top-level keys starting with x- that are ignored by Docker Compose and the Docker engine. You can use them to define composable service defintions by creating service fragments, and then mixing them in to create concrete services. Extensions can be used for any part of a Docker Compose file.