Running Magento2 in Kubernetes — Part 2: Building the Docker Image

Preface

This article explains how to build a Docker image from scratch or existing projects in three simple steps. It starts with an overview for readers whose daily work might not be related to build pipelines and deployment processes. For admins and seniors, who already opened their shell, jump to the “DevOps time” chapter to get the practical takeaway of the article.

Build and deploy

Docker has two critical phases: An image build phase and the startup of the container. Usually they are timely and physically separated. The build process runs on a build server (Gitlab, Jenkins). Generated images are pushed to a Docker registry (e.g. Gitlab, Google Container Registry or Docker Hub). On deployment they get pulled from the registry on the target machine.

What is a Magento build process?

Back in the days of Magento1 a build phase basically didn’t exist. Magento1 did everything for the developers (like merging CSS and JS files) on the fly. When Less/Sass became more popular some developers started to use Grunt to compile their CSS and minify JavaScript files. The cool kids even used Composer and Modman to install dependencies and libraries. Some of these steps could be already considered as “build phase”. They installed dependencies and prepared the code in the Git repository for optimal execution in a production environment.

When switching from developer to production mode (bin/magento deploy:mode:set production) at least setup:di:compile and setup:static-content:deploy get executed. The Magento DevDocs has a diagram which explains the detailed steps. It also mentions the configuration as an important ingredient for Magento’s build phase. In our approach the last step of the process „Check the compiled and static view files into repository” is the phase where the built Docker image is saved in the Docker registry.

How to configure your build phase?

Since Magento 2.1 a database connection isn’t necessary anymore to run SCD (static content deployment). This is possible by adding information about stores, settings and modules to the “shared configuration”, located at app/etc/config.php. In the build phase also Composer dependencies get resolved based on the composer.lock file.

There are several approaches to execute all these steps in an automated way. Some developers prefer Capistrano and other “zero downtime” scripts to execute the build phase on the target system before switching the symlink of the web root. Magento provides a tool which is rarely mentioned outside of the Magento Cloud documentation although it is available on Github already since three years: The Magento Commerce Cloud Deployment Tools, in short: ECE-Tools. With the 2002.1 release in February ECE-Tools introduced “scenarios” which allows easy customization and enhancements of each step. We highly recommend ECE-Tools as it provides much of what is needed to configure the Docker build and deploy process.

The ECE-Tools documentation might look a bit confusing at first, since it is woven into the Magento Cloud documentation, but it helps to understand how to configure themes and languages (see https://devdocs.magento.com/cloud/project/magento-env-yaml.html) and how to export your website and store configuration (see https://devdocs.magento.com/cloud/live/sens-data-over.html). But lean back, we’ll make the usage of ECE-Tools easy for you!

The deployment phase

For Magento2 there is one, maybe two important commands executed during deployment: The predominant is bin/magento setup:upgrade –keep-generated to update the database. The other one, which is maybe less known, is app:config:import. It imports the shared configuration file into the database which is very powerful if you want to maintain Magento’s configuration in Git. This command can even create new stores and websites – or remove them.

Putting these two commands in a Docker startup script could already do the job, but ECE-Tools does much more for you: If there are no tables in the database it starts Magento’s installation routine which sets a base domain, creates an admin user etc. It creates and updates the app/etc/env.php from environment variables. The command also flushes the cache and can take care of basic cache heating after the deployment has completed. For convenience there are lot of little commands executed, check the deploy scenario for details.

When executing this standard scenario on container start in an environment with multiple instances it can become troublesome: Because ECE-Tools enables the maintenance mode when starting the setup:upgrade command, your site might be unreachable several times. The maintenance mode puts a file in the var folder, which has to be shared across all Magento instances, and therefore shuts down all running instances at once. Flushing the cache every time the number of Magento instances is scaled up can become a nightmare on traffic peaks. We tried to balance consistency, zero-downtime and production issues in our modified scenarios, which will get explained in another article when talking about K8S deployment.

Project layout

Each command in a Dockerfile creates a new layer in the image’s filesystem. Docker caches them to re-use it on the next build if the files and the command hasn’t changed. Magento Commerce Cloud projects require to have the Magento source code in the Git repository root. To leverage Docker’s build cache on simple configuration changes it is beneficial to move config files out of the Magento root directory. We therefore recommend to move Magento to a “src” directory and keep the container and deployment related files outside of it.

Frankly, we use both project layouts for client projects, but the afore mentioned is recommended as the build time is much faster on simple K8S adjustments. You can keep the existing layout but will have to adjust some paths in the bundled files described in the next chapter.

DevOps time

After all the theory let’s get practical and see it all in action.

Step 1: Install Magento2 or prepare your existing project

First, install Magento via composer:

composer create-project --repository-url=https://repo.magento.com/
magento/project-community-edition src

The command will take several minutes. When finished Magento’s source code is installed in the src directory.

For existing projects move them to a src directory to follow the next steps in this tutorial:

export MAGENTO_ROOT="$(pwd)"
cd ..
mkdir magento-project
mv $MAGENTO_ROOT magento-project/src
cd magento-project
unset $MAGENTO_ROOT

Step 2: Install the Docker build tools

For convenience we bundled our Docker build tools in a Composer package. To install it switch to the src directory and execute:

composer require phoenix-media/magento2-cloud-build --no-update
composer update

This will install a Dockerfile, some helper scripts, basic configuration files (for nginx, PHP, Supervisord), Magento’s ECE-Tools and our enhanced ECE-Tools deployment scenarios. For package details visit https://github.com/PHOENIX-MEDIA/magento2-cloud-build.

Since Composer doesn’t execute shell commands automatically execute this command to copy the sample files to their target location:

vendor/bin/px-cloud-build-install

Your project root should now have the following file structure:
.dockerignore
.gitignore
Dockerfile
bin
docker
docker-compose.yml
helm
src

You most likely want to customize these files and src/bin/deploy for each project. The intention of the Composer package is to help you installing and update the files. A simple git diff will help you to identify necessary changes after you pulled a new version of the package and executed vendor/bin/px-cloud-build-install.

If you have an existing project it is recommended to do three things now:

1. Execute vendor/bin/ece-tools config:dump in an environment with a recent database. It will write store configuration and settings about SCD to app/etc/config.php. Add it to the Git repo. Having this information is important for the ECE-Tool’s build and deployment process.

2. Review src/.magento.env.yaml, especially the SCD_MATRIX regarding themes and languages. Without listing your theme, it won’t be generated during SCD execution which will cause errors in the frontend later.

3. Compare the .gitignore file in the Magento root with the one which just has been installed to the project root to make sure it fits your needs.

One important note: Although some developers might consider this as a sacrilege, at PHOENIX MEDIA we prefer to commit the vendor directory to the Git repo. This helps to trace all code changes on updates and it will also speed up build time of the Docker container. But it is just a matter of taste and philosophy. If you prefer to not have the Composer packages in Git just remove the exclamation mark in .gitignore for these two lines:

!/**/vendor
!/src/update/vendor

Step 3: Pack your bags

The Dockerfile uses phoenixmedia/nginx-php. It is an Alpine based Docker image which installs PHP, nginx, Postfix and Supervisord, to start all daemons in parallel. The image has only half the size of the official PHP image and supports PHP 7.2–8.2. You can change the PHP version by simply editing the image tag in the Dockerfile located in the project root.

Before you start the build in the next step make sure you have a local Docker instance running. We recommend to configure it for 4 cores and 4 GB memory — or more.

Now you should be good to build your first Magento container. You have the option to just build the container with “docker build .” or to directly build and run it with “docker-compose up”. Once finished check the running instance in the browser: http://localhost. Carefully read the output and verify there are no errors.

In case you need to adjust something just start over by executing “docker-compose down -v --rmi local” and “docker-compose up” again.

Environments and mysterious variables

When investigating the bundled docker-compose.yml file, you will notice that the container requires three environment variables:

- MAGENTO_CLOUD_ROUTES
- MAGENTO_CLOUD_VARIABLES
- MAGENTO_CLOUD_RELATIONSHIPS

They provide information for the “vendor/bin/ece-tools deploy” command to generate app/etc/env.php during container startup and configure the Magento2 base URL. You can google for the environment variables, but you won’t find much details about their values. The Magento Cloud documentation just gives a hint: It is a base64-encoded JSON object used to provide service configuration and credentials.

Use this command to decode the example value of MAGENTO_CLOUD_VARIABLES:

echo "yJBRE1JTl9GSVJTVE5BTUUiOiJZb3VyIEZpcnN0bmFtZSIsIkFETUlOX0xBU1ROQU1FIjoiWW91ciBMYXN0bmFtZSIsIkFETUlOX0VNQUlMIjoiYWNjb3VudEBtYWlsLmNvbSIsIkFETUlOX1VTRVJOQU1FIjoiYWRtaW5pc3RyYXRvciIsIkFETUlOX1BBU1NXT1JEIjoidG9wQFNlY3JlMSIsIkFETUlOX1VSTCI6ImFkbWluIiwiQ1JZUFRfS0VZIjoiNmZnMzdqYzUyOWUwZDlmMDY5NjY1Y2JjOWZlMTcwYjIifQ" | base64 -d

Output:

{
  "ADMIN_FIRSTNAME": "Your Firstname",
  "ADMIN_LASTNAME": "Your Lastname",
  "ADMIN_EMAIL": "account@mail.com",
  "ADMIN_USERNAME": "administrator",
  "ADMIN_PASSWORD": "top@Secre1",
  "ADMIN_URL": "admin",
  "CRYPT_KEY": "6fg37jc529e0d9f069665cbc9fe170b2"
}

And another one for MAGENTO_CLOUD_RELATIONSHIPS:

echo "eyJkYXRhYmFzZSI6W3siaG9zdCI6Im15c3FsIiwicGF0aCI6Im1hZ2VudG8iLCJwYXNzd29yZCI6InRvcFNlY3JldCIsInVzZXJuYW1lIjoibWFnZW50byIsInBvcnQiOiIzMzA2In1dLCJyZWRpcyI6W3siaG9zdCI6InJlZGlzIiwgInBvcnQiOiAiNjM3OSJ9XSwiZWxhc3RpY3NlYXJjaCI6W3siaG9zdCI6ImVsYXN0aWNzZWFyY2giLCAicG9ydCI6ICI5MjAwIiwgInNjaGVtZSI6Imh0dHAifV0sInJhYmJpdG1xIjpbeyJob3N0IjoicmFiYml0bXEiLCAidXNlcm5hbWUiOiJwaG9lbml4IiwgInBhc3N3b3JkIjoiTWFnZW50bzEyMyIsICJzY2hlbWUiOiJhbXFwIiwgInBvcnQiOiAiNTY3MiJ9XX0=" | base64 -d

Output:

{
  "database": [
    "host": "mysql",
    "path": "magento",
    "password": "topSecret",
    "username": "magento",
    "port": "3306"
  }],
  "redis": [{
    "host": "redis",
    "port": "6379"
  }],
  "elasticsearch": [{
    "host": "elasticsearch",
    "port": "9200",
    "scheme": "http"
  }],
  "rabbitmq": [{
    "host": "rabbitmq",
    "username": "phoenix",
    "password": "Magento123",
    "scheme": "amqp",
    "port": "5672"
  }]
}

To set the base URL for Magento2 modify the MAGENTO_CLOUD_ROUTES:

echo "eyJodHRwOlwvXC9sb2NhbGhvc3RcLyI6eyJ0eXBlIjoidXBzdHJlYW0iLCJvcmlnaW5hbF91cmwiOiJodHRwOlwvXC97ZGVmYXVsdH1cLyJ9fQ==" | base64 -d

Output:

{
  "http://localhost/": {
    "type": "upstream",
    "original_url": "http://{default}/"
  }
}

Replace “http://localhost” with your project domain. Hint: It is fine to use “http” instead of “https”. There has been some issues in older ece-tools versions.

You can modify these values and base64 encode them again. Those values will be later stored in a Kubernetes Secret and injected into the pod.

What’s next?

After everything is working commit all files to Git. To trigger the Docker build process on every commit you should include it in your build pipeline. The image can then be pushed to a container registry.

A simple example of how to set up a Github Action pipeline checkout the magento2-build repository. The pipeline performs all described steps above and pushes the Magento2 image to DockerHub.

Gitlab users find a great tutorial in the documentation. Google Cloud users can follow this steps: https://cloud.google.com/cloud-build/docs/building/build-containers.

From there we’ll continue in the next article and lay the foundation for the Kubernetes deployment. Meanwhile get familiar with some K8S concepts and evaluate options to setup a K8S cluster.

Summary

Magento’s ECE-Tools is a set of powerful modules to streamline build and deployment processes. It has a steep learning curve, but it is recommended to get familiar with it if you plan work with Magento Commerce Cloud or the approach described in this series.

The magento2-cloud-build Composer package provides all necessary files to build and run existing projects in a Docker container. The build and deployment will require adjustments to meet project requirements.

We appreciate your valuable feedback and looking forward to contributions to our Github repos.

Continue reading:

• Running Magento2 in Kubernetes —
  Part 1: Basic concepts and best practices
• Running Magento2 in Kubernetes —
  Part 2: Building the Docker image (this article)
• Running Magento2 in Kubernetes —
  Part 3: Deploying the application