Single-line Onboarding Using Docker Compose

How to speed up engineering onboarding with this one weird trick!

Breeze through onboarding by using Docker locally

All my career, I’ve worked at various startups that have had a very similar onboarding strategy. Because the company (and product) is small enough, it’s reasonable to ask the engineers to run the entirety of the product locally on their machine for development. If this sounds familiar, the following steps might also sound familiar:

1. Download the (correct version) of the backend language
2. Download the (correct version) of NPM/Node for the frontend
3. Download the package manager for the backend
4. Download the package manager for the frontend
5. Download the (correct version) of the DB to run locally
6. (optionally) download the other non-primary storage DBs (e.g. ElasticSearch, Redis)
7. Install the frontend and backend dependencies
8. Start the frontend server
9. Start the backend server
10. Start other backend processes (e.g. async workers), if needed
11. Debug failures and undocumented dependencies and return to step 6

This process is tiring, but expected. Setting up a development environment is something that takes between a few hours and a few days and should not have to be repeated. Unfortunately, it’s also difficult to help others because computers, languages and versions change over time so they may run into errors you didn’t encounter or you might have forgotten the errors you did encounter when you set up your environment 6 months prior. Surely there is a better way.

The Shortcut: Docker Compose

About a year ago I took a course on Udemy that covered Docker and deployment orchestration. While I’ve used Docker pretty extensively in my roles, it’s almost always been for the purpose of managing deployed environments (typically via AWS ECS).

Docker Compose was new to me, and I hadn’t tried to learn much about it before the course. I found the official documentation daunting and too unclear to sell me on its benefits. The course, however, quickly displayed how powerful Docker Compose could be with a relatively simple YAML file.

Instead of needing to install the various dependencies on my host machine, I could define a few containers with the same versions of libraries and languages used in production. I could then run one command to spin up all the containers that I needed:

docker compose up

The result would be a terminal shell that shows the output of building several images and running them as containers. At Singularity, we have quite a few services to run locally for a single application, so here is an example of one of our docker-compose.yml files:

version: '3.7'

services:
  db:
    image: postgres:12.11
    restart: always
    environment:
      - POSTGRES_PASSWORD=app_db_password
      - POSTGRES_USER=app_db_user
      - POSTGRES_DB=app_db
    ports:
      - 5432:5432
    volumes:
      - db-data:/var/lib/postgresql/data

  app_api:
    restart: always
    build:
      context: ./backend
    env_file: secrets.env
    volumes:
      - ./backend:/usr/src/app/
    ports:
      - 5010:5010
    entrypoint: flask
    command:
      - run
      - --host
      - '0.0.0.0'
    environment:
      - 'MQ_BROKER_URL=amqp://admin:mypass@rabbitmq:5672'
      - FLASK_APP=autoapp.py
      - FLASK_RUN_PORT=5010
      - DATABASE_URI=postgresql://app_db_user:app_db_password@db/app_db
      - FLASK_DEBUG=1
    depends_on:
      - db

  frontend:
    restart: always
    build:
      context: ./frontend
    volumes:
      - ./frontend:/usr/src/app/
    ports:
      - 3000:3000
    entrypoint: yarn
    command:
      - start
    depends_on:
      - app_api

  rabbitmq:
    restart: always
    image: rabbitmq:3-management
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
        - 5673:5672
        - 15673:15672
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

  worker:
    restart: always
    build:
      context: ./backend
    env_file: secrets.env
    volumes:
      - ./backend:/usr/src/app/
    entrypoint: celery
    command:
      - -A
      - worker.tasks
      - worker
      - -Q
      - worker-queue
    environment:
      - 'MQ_BROKER_URL=amqp://admin:mypass@rabbitmq:5672'
      - 'SERVICE_API_HOST=http://app_api:5010'
    depends_on:
      - rabbitmq
      - app_api

volumes:
  db-data:
  rabbitmq-data:

You might have noticed something interesting with the URLs: Docker Compose will automatically create a Docker bridge network and attach all the services (“app_api”, “worker”, etc.) to the network. Additionally, the containers attached to that network will have a network alias that is the service name. This is why the hostname of the DATABASE_URI is just db , for example.

Expanding on it

You might later want to add something to your infrastructure. For example, we wanted to try adding tracing via OpenTelemetry using Jaeger as the backend. To do this, we instrumented our code according to the OpenTelemetry docs, and we added Jaeger as a service in our docker-compose.yml :

diff --git a/docker-compose.yml b/docker-compose.yml
index 9dc64ca..4f1d459 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -112,6 +112,14 @@ services:
    depends_on:
      - rabbitmq
      - app_api
 
+  jaeger:
+    restart: always
+    image: jaegertracing/all-in-one:1.44
+    ports:
+      - 16686:16686
+      - 4317:4317
+    environment:
+      - COLLECTOR_OTLP_ENABLED=true
 
 volumes:
   db-data:

Adding the code was still tedious, but adding this one service for Docker Compose allowed me to test the OTel implementation without needing to deploy anything (even to the development environment)!

Seeing an extra component of infrastructure being added this easily made me feel unencumbered to try more things: cacheing via Redis, search via ElasticSearch, even trying out a timeseries database instead of PostgreSQL.

The Caveats

Of course, nothing is simple and that includes changing a local development environment. Of the 11 steps that I highlighted above, the first 4 are still required to have any sort of integrated development environment (where you want your code editor to resolve the installed libraries and help with text completion and code suggestions).

Additionally, developers will have to download docker and have some (at least rudimentary) understanding of using docker and docker compose in order to fluently manage their containers. This might seem reasonable for a backend, full stack, or devops engineers, but could be asking a lot of a frontend engineer.

Ultimately, don’t change your tooling just to change it, but carefully understand what it would mean to do so. Decide with your team if it’s worth it. Perhaps try it out with yourself and a couple others before rolling it out more broadly.

Interested in this type of work? Have suggestions for how we could do it better? Email us! We’re hiring 🌱