Docker Compose from development to production

Docker Compose is an amazing tool to create the development environment for your application stack. It allows you to define each component of your application following a clear and simple syntax in YAML files.

With the introduction of the docker compose v3 definition these YAML files are ready to be used directly in production when you are using a Docker Swarm cluster.

But does this mean that you can reuse the same compose file for development and production? Even reuse the same compose file for staging? Well, more or less, but for get it we’ll need:

• Variable interpolation: use environment variables for some values that change on each environment.
• Override configuration: You can define a second (or more) compose files that just change something from the first and compose will take care of merge both files.

Differences between compose files for dev and prod

When you are in development you’ll probably want to check your code changes in real-time. The common way to do this is mounting a volume with your source code in the container that has the runtime of your application. But for production this works differently.

In production you have a cluster with many nodes, and a volume is local to the node where your container (or service) is running, then you cannot mount the source code without complex stuff that involve code synchronization, signals, etc.

Instead of this we usually want to build an image with the specific version of your code inside, and it’s a convention to tag individually each version (semantic versioning or other system of your preference).

Override configuration

With this in mind and that probably your dependencies could be different in dev/prod it’s clear that we’ll need different configuration files.

Docker compose supports the concatenation of different compose files to get a final configuration file with the configuration merged. I can show you how this works with an example:

$ cat docker-compose.yml
version: "3.2"

services:
  whale:
    image: docker/whalesay
    command: ["cowsay", "hello!"]
$ docker-compose up
Creating network "composeconfigs_default" with the default driver
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ________
whale_1  | < hello! >
whale_1  |  --------
whale_1  |     \
whale_1  |      \
whale_1  |       \
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        \______ o          __/
whale_1  |         \    \        __/
whale_1  |           \____\______/
composeconfigs_whale_1 exited with code 0

As I said compose supports merging more than one compose files, this allows me to override anything in a secondary file. For example:

$ cat docker-compose.second.yml
version: "3.2"
services:
  whale:
    command: ["cowsay", "bye!"]

$ docker-compose -f docker-compose.yml -f docker-compose.second.yml up
Creating composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | < bye! >
whale_1  |  ------
whale_1  |     \
whale_1  |      \
whale_1  |       \
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        \______ o          __/
whale_1  |         \    \        __/
whale_1  |           \____\______/
composeconfigs_whale_1 exited with code 0

This syntax is not so comfortable for development, where you’ll probably execute this command many times.

Fortunately compose is automatically searching for an specific file named docker-compose.override.yml for overriding the docker-compose.yml values. If I rename the second file I’ll get the same result with just the initial command:

$ mv docker-compose.second.yml docker-compose.override.yml
$ docker-compose up
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1  |  ______
whale_1  | < bye! >
whale_1  |  ------
whale_1  |     \
whale_1  |      \
whale_1  |       \
whale_1  |                     ##        .
whale_1  |               ## ## ##       ==
whale_1  |            ## ## ## ##      ===
whale_1  |        /""""""""""""""""___/ ===
whale_1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
whale_1  |        \______ o          __/
whale_1  |         \    \        __/
whale_1  |           \____\______/
composeconfigs_whale_1 exited with code 0

Good, simple to remember again.

Variable interpolation

Compose files support variable interpolation, and support default values. This means that you can do something like:

services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
...

And if you execute the docker-compose build (or push) without the $MY_SERVICE_VERSION environment variable, its value will be latest, but if you assign a value to it before the build it will be used when build or pushed to the private.registry.mine registry.

My Mantra

Something that is working for my use cases could cover yours as well. I’m following this simple rules:

• All my stacks for production, development (or other environments) are defined via docker-compose files.
• The compose files needed to cover all my environments are avoiding as much as possible configuration duplication.
• I just need one simple command to work on each environment.
• Define your productive configuration in the docker-compose.yml file.
• Define environment variables to identify image tags or other variables that could change for any environment (staging, integration, production).
• Use the production values as defaults, this minimizes impact in case you launch the stack in production without an expected variable.
• In production deploy with the docker stack deploy --compose-file docker-compose.yml --with-registry-auth my-stack-name command.
• Launch your development environment with docker-compose up -d.

Let’s see it with a simple example.

# docker-compose.yml
...
services:
  my-service:
    build:
      context: .
    image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
    environment:
      API_ENDPOINT: ${API_ENDPOINT:-https://production.my-api.com}
...

And

# docker-compose.override.yml
...
services:
  my-service:
    ports: # This is needed for development!
      - 80:80
    environment:
      API_ENDPOINT: https://devel.my-api.com
    volumes:
      - ./:/project/src
...

I can use the docker-compose command (docker-compose up) to launch my stack in dev mode, with the source code mounted in /project/src.

I can use this same configuration files in production! And I could reuse exactly the same docker-compose.yml file for staging. To deploy this into production I just need to build and push the image with a predefined tag in my CI stage:

export MY_SERVICE_VERSION=1.2.3
docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml push

In production this can be launched with just the commands below:

export MY_SERVICE_VERSION=1.2.3
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

And if you want to do the same in staging it’s as easy as to define the environment variable needed for staging:

export MY_SERVICE_VERSION=1.2.3
export API_ENDPOINT=http://staging.my-api.com
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth

With all these things into consideration we have used docker compose files, two as much in my examples and without duplicate configurations, to any of your environments!