Docker Compose Demystified: A Comprehensive Guide to Efficient Container Management, Deployment and Scaling Your Containers

13 min readMar 23, 2023

Difference Between Docker and Docker-Compose:

Docker and Docker Compose are both tools used for managing and deploying containerized applications, but they serve different purposes.

Docker:

Docker is a tool used to build, run, and manage individual containers. It provides a way to package an application and its dependencies into a single container that can be run on any system that supports Docker. Docker makes it easy to deploy and scale applications by allowing you to create, run, and manage containers from a single interface.

Some of the key features of Docker include:

Image-based deployment: Docker images are used to package an application and its dependencies into a single, portable package.
Containerization: Docker containers provide an isolated environment for running applications, making it easy to deploy and scale applications.
Resource isolation: Docker uses resource constraints to limit the amount of CPU, memory, and I/O that a container can use.
Networking: Docker provides a way to connect containers together using virtual networks, making it easy to create complex applications.

Docker Compose:

Docker Compose is a tool used to manage multiple containers as a single application. It provides a way to define and manage a multi-container Docker application using a YAML file. With Docker Compose, you can define the services that make up your application and their dependencies, and then deploy and manage them as a single unit.

Some of the key features of Docker Compose include:

Multi-container deployment: Docker Compose allows you to define and deploy multiple containers as a single unit.
Service definitions: Docker Compose provides a way to define the services that make up your application and their dependencies.
Networking: Docker Compose provides a way to create custom networks for your containers and services.
Volume management: Docker Compose allows you to manage the volumes used by your containers and services.

Why use Docker Compose?

Docker Compose offers a range of benefits to developers. Some of the key benefits include:

Simplified Configuration: Docker Compose allows developers to define the entire application’s configuration in a single file, making it easier to manage complex applications, especially those with multiple containers.
Reproducible Builds: With Docker Compose, developers can ensure that the same configuration is used across all environments, from development to production, eliminating issues caused by differences between environments.
Easy Scaling: Docker Compose makes it easy to scale services up or down based on the application’s needs.
Improved Collaboration: Docker Compose allows developers to work together on the same application, with the same configuration, regardless of their development environment.

Installing Docker Compose

The installation process for Docker Compose varies depending on your operating system. The official Docker documentation provides detailed instructions on how to install Docker Compose for various platforms, including Linux, macOS, and Windows. Here are the steps for installing Docker Compose on a Linux system:

Open a terminal window on your Linux system.
Update the package index:

sudo apt update

Install Docker Compose:

sudo apt install docker-compose

Verify that Docker Compose was installed correctly:

docker-compose --version

The following output was shown for the command above.

ubuntu@instance-20221121-0031:~$ docker-compose --version
docker-compose version 1.29.2, build unknown

Docker Compose file structure

The Docker Compose file is a YAML file that defines the services, networks, and volumes required for the application. The file’s structure is divided into the following sections:

version: This section specifies the version of the Docker Compose file syntax being used.
services: This section defines the services needed for the application, including their configuration and dependencies.
networks: This section defines the networks needed for the application.
volumes: This section defines the volumes needed for the application.

Here is an example of a basic Docker Compose file:

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
  db:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: mypassword
networks:
  frontend:
  backend:
volumes:
  data:

In this example, we define two services, web and db. The web service is based on the latest version of the Nginx image and exposes port 80. The db service is based on the latest version of the MySQL image and sets the MYSQL_ROOT_PASSWORD environment variable. We also define two networks, frontend and backend, and a volume called data.

Services in Docker Compose

Services are the building blocks of Docker Compose applications. Each service is defined in a YAML file and represents a container that provides a specific functionality. Services can be connected together using virtual networks, allowing you to create complex applications.

Here is an example of defining a service in Docker Compose:

version: '3'
services:
  web:
    build: .
    ports:
      - "5000:5000"

In this example, we define a service called “web” that is built using the Dockerfile in the current directory. We also map port 5000 on the host to port 5000 in the container.

Defining Containers in Docker Compose

Containers are the core building blocks of any Docker application. Docker Compose provides a simple and effective way to define the containers needed for your application in a single file. To define a container in Docker Compose, you need to specify its image, ports, volumes, and other configuration options.

Here is an example of defining a container in Docker Compose:

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./app:/usr/share/nginx/html

In this example, we define a service called “web” that uses the latest version of the Nginx image. We expose port 80 on the container and mount the local “./app” directory to the “/usr/share/nginx/html” directory inside the container.

Environment Variables in Docker Compose

Environment variables are used to pass configuration information to containers at runtime. Docker Compose makes it easy to define environment variables for your containers using the “environment” key.

Here is an example of defining environment variables in Docker Compose:

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    environment:
      - ENVIRONMENT=production

In this example, we define a service called “web” that uses the latest version of the Nginx image. We expose port 80 on the container and define an environment variable called “ENVIRONMENT” with a value of “production”.

Networking in Docker Compose

Networking is an important aspect of any Docker application. Docker Compose provides a powerful networking feature that allows you to define and manage the network infrastructure of your application.

Here is an example of defining a network in Docker Compose:

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    networks:
      - frontend
  db:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: mypassword
    networks:
      - backend
networks:
  frontend:
  backend:

In this example, we define two services, “web” and “db”. We define two networks, “frontend” and “backend”. We connect the “web” service to the “frontend” network and the “db” service to the “backend” network.

Volumes in Docker Compose

Volumes are used to persist data generated by containers. Docker Compose allows you to define volumes for your containers using the “volumes” key.

Here is an example of defining a volume in Docker Compose:

version: '3'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - data:/usr/share/nginx/html
volumes:
  data:

In this example, we define a service called “web” that uses the latest version of the Nginx image. We expose port 80 on the container and mount a volume called “data” to the “/usr/share/nginx/html” directory inside the container.

Writing a Docker Compose File to Deploy My Static Website

version: '3'
services:
  web:
    image: my-website
    ports:
      - "80:80"
    volumes:
      - /home/ubuntu/Desktop/Docker-Compose/web_volume:/usr/share/nginx/html

In this file, we define a service named "web" that uses the "my-website" Docker container image. We map port 80 in the container to port 80 on the host machine, so that the website can be accessed from a web browser. We also define a volume that maps the HTML folder in the container to the "/home/ubuntu/Desktop/Docker-Compose/web_volume" directory on the host machine. This allows any changes made to the website files in the container to be persisted on the host machine.

To use this docker-compose.yml file, simply save it to a directory on your machine and run the command "docker-compose up" in that directory. This will start the "web" service and map the container's port 80 to the host machine's port 80, as well as mount the HTML folder in the container to the specified volume on the host machine.

Building Docker images with Docker Compose

Docker Compose provides a way to build Docker images using a Dockerfile. Here is an example of building a Docker image using Docker Compose:

version: '3'
services:
  web:
    build: .
    image: my-web-app:latest

In this example, we define a service called “web” that is built using the Dockerfile in the current directory. We also specify a custom image name using the “image” key.

To build the image, run the following command:

docker-compose up --build

The following is the output of the command.


root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker-compose up --build
Recreating docker-compose_web_1 ... done
Attaching to docker-compose_web_1
web_1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web_1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web_1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web_1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web_1  | /docker-entrypoint.sh: Configuration complete; ready for start up
web_1  | 2023/03/22 19:23:09 [notice] 1#1: using the "epoll" event method
web_1  | 2023/03/22 19:23:09 [notice] 1#1: nginx/1.23.3
web_1  | 2023/03/22 19:23:09 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
web_1  | 2023/03/22 19:23:09 [notice] 1#1: OS: Linux 5.15.0-1030-oracle
web_1  | 2023/03/22 19:23:09 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
web_1  | 2023/03/22 19:23:09 [notice] 1#1: start worker processes
web_1  | 2023/03/22 19:23:09 [notice] 1#1: start worker process 29
web_1  | 2023/03/22 19:23:09 [notice] 1#1: start worker process 30
web_1  | 206.84.143.45 - - [22/Mar/2023:19:23:26 +0000] "GET / HTTP/1.1" 200 3247 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" "-"

Scaling Services in Docker Compose

Scaling is an important aspect of any containerized application. Docker Compose provides a simple and effective way to scale your services up or down depending on your needs. To scale a service in Docker Compose, you can use the “scale” command.

Here is an example of scaling a service in Docker Compose:

docker-compose up -d

The following is the output of the above command.


root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker-compose up -d
Stopping and removing docker-compose_web_2 ... done
Stopping and removing docker-compose_web_3 ... done
Starting docker-compose_web_1              ... done
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS                PORTS                                       NAMES
73c06980dbd1   my-website                       "/docker-entrypoint.…"   4 minutes ago   Up About a minute     0.0.0.0:80->80/tcp, :::80->80/tcp           docker-compose_web_1
984c2e5899b4   gcr.io/cadvisor/cadvisor-arm64   "/usr/bin/cadvisor -…"   2 days ago      Up 2 days (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor

docker-compose scale web=3

The following is the output error when we try to scale up my web instance.


root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker-compose scale web=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
WARNING: The "web" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating docker-compose_web_2 ...
Creating docker-compose_web_2 ... error
WARNING: Host is already in use by another container
Creating docker-compose_web_3 ... error

ERROR: for docker-compose_web_2  Cannot start service web: driver failed programming external connectivity on endpoint docker-compose_web_2 (e9debb4520d657e9e70a473cd71d81f0add8d12921978096e99b572743a60144): Bind for 0.0.0.0:80 failed: port is already allocated

ERROR: for docker-compose_web_3  Cannot start service web: driver failed programming external connectivity on endpoint docker-compose_web_3 (1e34ba746acc197493c195b30d617f197367adcf7a03e8e2e48acd82132771e5): Bind for 0.0.0.0:80 failed: port is already allocated
ERROR: Cannot start service web: driver failed programming external connectivity on endpoint docker-compose_web_2 (e9debb4520d657e9e70a473cd71d81f0add8d12921978096e99b572743a60144): Bind for 0.0.0.0:80 failed: port is already allocated
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose#

The error message indicates that the user tried to scale up the number of instances of the “web” service in Docker Compose but failed because the port specified for the service on the host is already being used by another container. The driver failed to program external connectivity on the endpoints for the two additional instances of the “web” service because the port 80 on the host is already allocated to another container or process. The user needs to resolve the port conflict before attempting to scale up the number of instances of the “web” service.

This problem can be solved by avoiding port clashes using the following docker compose file configuration.


version: '3'
services:
  web:
    image: my-website
    ports:
      - 80

Now when the command docker-compose scale web=3 is run, the operation is successful and the following output is shown.


root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker-compose scale web=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting docker-compose_web_1 ... done
Creating docker-compose_web_2 ... done
Creating docker-compose_web_3 ... done
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                PORTS                                       NAMES
f0c5dd66ef91   my-website                       "/docker-entrypoint.…"   6 seconds ago    Up 5 seconds          0.0.0.0:49156->80/tcp, :::49156->80/tcp     docker-compose_web_3
0b12eef6d228   my-website                       "/docker-entrypoint.…"   6 seconds ago    Up 5 seconds          0.0.0.0:49155->80/tcp, :::49155->80/tcp     docker-compose_web_2
361923fe8316   my-website                       "/docker-entrypoint.…"   49 seconds ago   Up 6 seconds          0.0.0.0:49154->80/tcp, :::49154->80/tcp     docker-compose_web_1
984c2e5899b4   gcr.io/cadvisor/cadvisor-arm64   "/usr/bin/cadvisor -…"   2 days ago       Up 2 days (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor

In this case the web app has successfully scaled up and is access on the ports 49154–49154 . As shown in the screenshot attached below.

Now let us do one more interesting task to do. I’ll write a Docker Compose YAML file, I will this time also bind the volume, so when I change the content of a single folder in the host machine the content of all the containers change.

The YAML file is as following.


version: '3'
services:
  web:
    image: my-website
    ports:
      - 80
    volumes:
      - /home/ubuntu/Desktop/Docker-Compose/web_volume:/usr/share/nginx/html

When I ran the command docker-compose scale web=3 the following output is shown.


root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker-compose scale web=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting docker-compose_web_1 ... done
Creating docker-compose_web_2 ... done
Creating docker-compose_web_3 ... done

root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose# docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS                PORTS                                       NAMES
33ce9ea2fb67   my-website                       "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes          0.0.0.0:49160->80/tcp, :::49160->80/tcp     docker-compose_web_2
3f6a0cef9b70   my-website                       "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes          0.0.0.0:49159->80/tcp, :::49159->80/tcp     docker-compose_web_3
7c0ff920c2a3   my-website                       "/docker-entrypoint.…"   5 minutes ago   Up 4 minutes          0.0.0.0:49158->80/tcp, :::49158->80/tcp     docker-compose_web_1
984c2e5899b4   gcr.io/cadvisor/cadvisor-arm64   "/usr/bin/cadvisor -…"   3 days ago      Up 3 days (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor

Now since we have 3 containers running in the Docker, I’ll change the index.html in the volume bind and observe if the changes are reflected in all running containers.

The process is shown as following.



root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# docker-compose ps
        Name                      Command               State                   Ports
------------------------------------------------------------------------------------------------------
docker-compose_web_1   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:49158->80/tcp,:::49158->80/tcp
docker-compose_web_2   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:49160->80/tcp,:::49160->80/tcp
docker-compose_web_3   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:49159->80/tcp,:::49159->80/tcp
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# cat index.html
Hello
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# echo "New Word" >> index.html

root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# curl http://127.0.0.1:49158/
Hello
New Word
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# curl http://127.0.0.1:49159/
Hello
New Word
root@instance-20221121-0031:/home/ubuntu/Desktop/Docker-Compose/web_volume# curl http://127.0.0.1:49160/
Hello
New Word

As we can see in the output above, the changes are reflected in the all the running containers.

Docker Compose Commands and Usage:

Docker Compose provides a set of commands to manage your containers and services. Here are some of the most commonly used commands:

up: Creates and starts the containers defined in your Docker Compose file.
down: Stops and removes the containers defined in your Docker Compose file.
build: Builds or rebuilds the Docker images defined in your Docker Compose file.
start: Starts the containers defined in your Docker Compose file.
stop: Stops the containers defined in your Docker Compose file.
ps: Lists the running containers defined in your Docker Compose file.

Here is an example of using these commands:

$ docker-compose up -d
$ docker-compose ps
$ docker-compose stop
$ docker-compose down

In this example, we start the containers defined in our Docker Compose file using the “up” command, list the running containers using the “ps” command, stop the containers using the “stop” command, and finally remove the containers using the “down” command.

Best Practices for Using Docker Compose

Here are some best practices to follow when using Docker Compose:

Keep your Docker Compose file in version control to track changes.
Use descriptive names for your services and containers.
Use environment variables to configure your containers.
Use volumes to persist data generated by your containers.
Use the “depends_on” key to define dependencies between services.
Use healthchecks to monitor the health of your containers.
Use the “restart” key to automatically restart failed containers.

Troubleshooting Docker Compose

Docker Compose provides a powerful toolset for debugging and troubleshooting issues with your containers and services. Here are some tips to help you troubleshoot issues with Docker Compose:

Check the logs of your containers using the “logs” command.
Use the “exec” command to run commands inside your containers.
Use the “ps” command to check the status of your containers.
Use the “inspect” command to view detailed information about your containers.
Check the health of your containers using healthchecks.
Check for conflicts with other containers or services running on your system.

Alternatives to Docker Compose

Alternatives to Docker Compose:

While Docker Compose is a popular tool for managing Docker applications, there are other alternatives available that may better suit your needs.

Some popular alternatives include:

Kubernetes: A container orchestration tool that provides advanced features for managing containerized applications.
Docker Swarm: A native clustering and orchestration tool for Docker that provides a simple and scalable way to deploy containers.
Nomad: A distributed system for managing containerized and non-containerized applications.

In conclusion, Docker and Docker Compose are essential tools for modern application development, and understanding their features and differences is crucial for efficient container management and deployment. Docker provides a platform for containerization, while Docker Compose allows developers to manage multiple containers as services.

Through this blog, we explored various topics related to Docker Compose, such as its installation, file structure, services, defining containers, working with environment variables, networking, volumes, building Docker images, scaling services, commands and usage, best practices, troubleshooting, and alternatives. These topics offer developers a comprehensive overview of Docker Compose and can help them optimize their container management and deployment process.

I welcome you to connect with me on LinkedIn or reach out to me via email at saadrabbani123@gmail.com if you have any further questions or simply want to chat.