Improve Drupal 8 performance on Docker and OSX

This text is intend for anyone already have Drupal experience and use docker on localhost to run the project

It is a very knowing docker mount volume performance issue on OSX (https://medium.freecodecamp.org/speed-up-file-access-in-docker-for-mac-fbeee65d0ee7). This issue gather big impact when there are a plenty of files in your project.

To solve it, shall use docker-sync (https://github.com/EugenMayer/docker-sync). Basically, docker-sync creates a docker container and volume focused on sync files from your mac to inside service container (in our study field, Drupal). The sync process can be done by Rsync (https://rsync.samba.org/) or Unision (https://www.cis.upenn.edu/~bcpierce/unison/), famous tools to transfer files.

Now, let make your hands dirty.

The Big Picture

My Drupal files list is like that

There are some folder doesn’t belongs to Drupal default instalation, but don't worry about. Let’s focus on docroot and vendor folders, where our problem lives.

These two Drupal folders are very huge and because of them, the docker performance is very poor. In the vendor are placed composer installed library. In the docroot are placed the Drupal files ( read by Webserver Apache/Nginx).

When you type http://localhost all files in docroot are required on Drupal bootstrap, as you probably notice, the response time is terrible, almost 1 minute per request.

Solution

1 — Install docker-sync

docker-sync is a Ruby gem project, so to install it you need the RubyGem (gem ) installed in your OSX. Overall, RubyGem is already native installed on OSX. To check it, open a terminal session and type the command gem -h , you should see something like that:

RubyGems is a sophisticated package manager for Ruby.  This is a
basic help message containing pointers to more information.
Usage:
gem -h/--help
gem -v/--version
gem command [arguments...] [options...]
Examples:
gem install rake
gem list --local
gem build package.gemspec
gem help install
Further help:
gem help commands list all 'gem' commands
gem help examples show some examples of usage
gem help gem_dependencies gem dependencies file guide
gem help platforms gem platforms guide
gem help <COMMAND> show help on COMMAND
(e.g. 'gem help install')
gem server present a web page at
http://localhost:8808/
with info about installed gems
Further information:
http://guides.rubygems.org

If you dont have RubyGem installed, please follow the instructions on https://rubygems.org/pages/download.

Backing to our case, open a terminal session and type gem install docker-sync

I strongly suggest to you read a little bit about docker-sync (https://docker-sync.readthedocs.io/en/latest/) before to continue to read this article. The docker-sync documentation is pretty simple, you will take no longer than 5 minutes.

2 — Configure docker-sync.yml

This file is used by docker-sync to find out what folders will be synced and some other rules. The docker-sync.yml file must be created in the root project

In general, for the docker-sync service, you only need to setup sync , sync_excludes and sync_userid parameters

version: "2"
syncs:
digital1st-backend-app-sync:
src: './'
sync_excludes: ['.git', '.docker-sync']
sync_userid: '502'

In the src you have to set what folder will be copied to into the container. In the sync_excludes what folders you wish to exclude. In the sync_userid is your OSX user. To find out what is your user_id, open a terminal session and type de command id -u .

3 — Dockerfile and Entrypoint

If you alread have a Dockerfile, you can just use this file as reference

FROM mobingi/ubuntu-apache2-php7:7.2
# Install dependencies.
# ---------------------
RUN apt-get update -y               \
&& apt-get install -y \
bash-completion \
build-essential \
curl \
gzip \
imagemagick \
libfontconfig1 \
libjpeg-dev \
libpng12-dev \
libpq-dev \
mysql-client \
netcat \
python-software-properties \
sudo \
ssh \
tig \
vim \
xz-utils \
php7.2-zip \
--no-install-recommends \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/* \
&& rm -rf /var/tmp/*
# Configure Composer related environment.
# ---------------------------------------
ENV COMPOSER_ALLOW_SUPERUSER 1
ENV COMPOSER_DISABLE_XDEBUG_WARN 1
#
#
# # Install Composer.
# # -----------------
#
RUN curl -sS https://getcomposer.org/installer | php -- --version=1.6.3 \
&& mv composer.phar /usr/local/bin/composer \
&& composer global require "hirak/prestissimo:^0.3"
# Build-time configuration.
# -------------------------
# This is mostly useful to override on CIs.
ARG APP_NAME=drupal
ARG GROUP_ID=1000
ARG USER_ID=1000
# Configure environment.
# ----------------------
ENV APP_NAME=${APP_NAME}
ENV GROUP_ID=${GROUP_ID}
ENV GROUP_NAME=${APP_NAME} USER_ID=${USER_ID} USER_NAME=${APP_NAME}
# Create group and user.
# ----------------------
RUN groupadd --gid ${GROUP_ID} ${GROUP_NAME}                                                                          \
&& echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers \
&& useradd -u ${USER_ID} -G users,www-data,sudo -g ${GROUP_NAME} -d /${APP_NAME} --shell /bin/bash -m ${APP_NAME} \
&& echo "secret\nsecret" | passwd ${USER_NAME}
# Apache2 configs.
# -------------
COPY ./drupal.conf /etc/apache2/sites-available/
RUN sudo a2dissite 000-default.conf && sudo a2ensite drupal
# Import files.
# -------------
COPY ./.bashrc /${APP_NAME}/.bashrc
RUN chown ${USER_NAME}:${GROUP_NAME} /${APP_NAME}/.bashrc
# Make site available.
# --------------------
RUN rm -Rf /var/www/drupal \
&& ln -s /${USER_NAME}/app/docroot /var/www/drupal
# Configure Node related environment.
# -----------------------------------
ENV NVM_DIR /usr/local/nvm
ARG VERSION=stable
ENV VERSION ${VERSION}
# Install Node.
# -------------
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash \
&& [ -s "$NVM_DIR/nvm.sh" ] \
&& . "$NVM_DIR/nvm.sh" \
&& nvm install ${VERSION} \
&& npm install -g yarn
# Setup NVM and Node sourcing.
# ----------------------------
RUN echo "\n# Source NVM scripts\nsource $NVM_DIR/nvm.sh" >> /etc/bash.bashrc
# Make sure any sudoer can control libs.
# --------------------------------------
RUN sudo chmod a+w -R /usr/local
# Install node global dependencies.
# ---------------------------------
COPY ./.nvmrc /etc/.nvmrc
RUN . /usr/local/nvm/nvm.sh && \
cd /etc/ && nvm install v8.9.4 && nvm alias default && nvm use default && \
npm install -g yarn
# Setup user and initialization directory.
# ----------------------------------------
USER ${USER_NAME}
WORKDIR /${USER_NAME}/app
# Setup PATH env variables.
# -------------------------
ENV PATH="/${USER_NAME}/app/vendor/bin:$PATH"
# Install prestissimo for faster composer installs.
# -------------------------------------------------
RUN sudo cp -R /root/.composer /${USER_NAME}/.composer \
&& sudo chown ${USER_NAME}:${GROUP_NAME} -R /${USER_NAME}/.composer
RUN sudo sed -i "s/memory_limit = 128M/memory_limit = 2048M/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/display_errors = Off/display_errors = On/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/display_startup_errors = Off/display_startup_errors = On/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT/error_reporting = E_ALL/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 99M/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/post_max_size = 8M/post_max_size = 100M/g" /etc/php/7.2/cli/php.ini \
&& sudo sed -i "s/max_execution_time = 30/max_execution_time = 300/g" /etc/php/7.2/cli/php.ini
RUN sudo sed -i "s/memory_limit = 128M/memory_limit = 2048M/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/display_errors = Off/display_errors = On/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/display_startup_errors = Off/display_startup_errors = On/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT/error_reporting = E_ALL/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 99M/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/post_max_size = 8M/post_max_size = 100M/g" /etc/php/7.2/apache2/php.ini \
&& sudo sed -i "s/max_execution_time = 30/max_execution_time = 300/g" /etc/php/7.2/apache2/php.ini
COPY ./entrypoint.sh /etc/entrypoint.sh
RUN sudo chmod +x /etc/entrypoint.sh
CMD ["/bin/bash"]
ENTRYPOINT ["/etc/entrypoint.sh"]

Basically, this Dockerfile install PHP dependencies to run Drupal, add drupal.conf (for apache), install Composer, install NodeJs and change some Apache configurations for easy debug and good performance.

The drupal.conf is :

<VirtualHost *:80>
ServerAdmin dev@mycompany.com
DocumentRoot /drupal/app/docroot
<Directory /drupal/app/docroot>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
<FilesMatch "(mobingi-init\.sh$|mobingi-install\.sh$)">
Require all denied
</FilesMatch>
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

At last, the entrypoint.sh :

#!/bin/bash
set -e
# Source NVM scripts
source /usr/local/nvm/nvm.sh
# Start services and loggers.
# ---------------------------
# sudo service php5-fpm restart > /tmp/php.log
sudo service apache2 restart > /tmp/apache2.log
# Await database.
# ---------------.
while ! nc -q 1 database-host 3306 </dev/null; do sleep 3; done
echo ""
echo "----------------------------- ---------"
echo "--------- Database connected! ---------"
echo "----------------------------- ---------"
echo ""
# Install Drupal, if not installed yet.
# -------------------------------------
#
if [ ! -f /drupal/app/docroot/sites/default/settings.local.php ]
then
sudo chown -R drupal:drupal /drupal/app
echo "# 1 - Run composer install."
sudo composer -n install --prefer-dist
echo "# 2 - Create basic files and ensure permissions."
mkdir -p /drupal/app/docroot/sites/default/files
echo "# 3 - Copy configuration files."
sudo cp /drupal/app/docroot/sites/template.settings.local.php /drupal/app/docroot/sites/default/settings.local.php
sudo chmod -R 777 /drupal/app/docroot/sites/default/settings.local.php
echo "# 4 - Configure database connection based on docker-compose env variables."
sed -i "s/{MYSQL_DATABASE}/${MYSQL_DATABASE}/g" /drupal/app/docroot/sites/default/settings.local.php
sed -i "s/{MYSQL_PASSWORD}/${MYSQL_PASSWORD}/g" /drupal/app/docroot/sites/default/settings.local.php
sed -i "s/{MYSQL_USER}/${MYSQL_USER}/g" /drupal/app/docroot/sites/default/settings.local.php
sed -i "s/{CONFIG_SYNC_DIRECTORY}/\/drupal\/app\/config\/default/g" /drupal/app/docroot/sites/default/settings.local.php
echo "# 5 - disabled sites/default/disabled_services.yml temporarily"
sudo mv -f /drupal/app/docroot/sites/default/services.yml /drupal/app/docroot/sites/default/disabled_services.yml
echo "# 6 - Set PHP_OPTIONS environment variable to fix sendmail error and run drush si thunder."
sudo chown -R drupal:drupal /drupal/app
cd /drupal/app/docroot
sudo /usr/bin/env PHP_OPTIONS="-d sendmail_path=`which true`" ../bin/drush si thunder install_configure_form.enable_update_status_module=NULL install_configure_form.enable_update_status_emails=NULL --site-name="Digital First" --account-name="admin" --account-pass="password" -y
echo "# 7 - set is_installing.txt file"
sudo touch /drupal/app/docroot/sites/default/is_installing.txt
echo "# 8 - set system.site."
sudo chmod -R 777 /drupal/.drush/cache/default
sudo chmod -R 777 /drupal/.drush/cache/usage
if [ -f /drupal/app/site-id ]
then
../bin/drush cset system.site uuid "`cat /drupal/app/site-id`" -y
fi
echo "# 9 - enable sites/default/disabled_services.yml"
sudo mv -f /drupal/app/docroot/sites/default/disabled_services.yml /drupal/app/docroot/sites/default/services.yml
echo "# 10 - set vcs config_directories and cache in settings.php"
sed -i "s/VCSconfig_directories/config_directories/g" /drupal/app/docroot/sites/default/settings.local.php
sed -i "s/CACHEsettings/settings/g" /drupal/app/docroot/sites/default/settings.local.php
echo ""
echo "--------------------------------------"
echo "--- SUCCESS!! \o/ ---"
echo "--------------------------------------"
else
#
#
# # Ensure permissions are correct.
# # -------------------------------
#
sudo chmod -R 777 /drupal/app/docroot/sites/default/files
sudo chmod 777 /drupal/app/docroot/sites/default/settings.local.php
sudo chmod +w -R /drupal/app/docroot/sites/default
echo ""
echo "--------------------------------------"
echo "--- Virtual Machine ready to work! ---"
echo "--------------------------------------"
echo ""
echo "Access your Drupal site at http://$(hostname -i)"
echo ""
exec "$@"
fi

At the first time you install the project are installed all of you need to run the project. Certainly your notice that the entrypoint runs some Thunder commands. Thunder is a Drupal distribution for professional publishing (https://www.drupal.org/project/thunder). If your Drupal project doesn't based on Thunder, please remove #6 — Set PHP_OPTIONS environment variable to fix sendmail error and drush si thunder

Futhermore, the entrypoint leaves a container session open when the project already installed. It is helpful because you only can run Drupal commands (drush, composer, etc) inside the container.

4 — Docker compose

There are 3 docker-compose files:

#BASE
version: '2'
services:
meta:
image: busybox # Necessary to make this container valid.
environment:
- MYSQL_USER=drupal
- MYSQL_DATABASE=drupal
- MYSQL_PASSWORD=password
- MYSQL_ROOT_PASSWORD=password
app:
extends: meta
image: digital1st-backend-app
container_name: digital1st-backend-app
build:
context: .
args:
USER_ID: "$USER_ID"
GROUP_ID: "$GROUP_ID"
hostname: app
ports:
- 80:80
cap_add:
- ALL
links:
- database:database-host
- memcached:memcached-host
volumes:
- $HOME/.ssh:/drupal/.ssh
- $HOME/.gitconfig:/drupal/.gitconfig
- $PWD:/drupal/app
memcached:
image: memcached:alpine
command: memcached -m 64
ports:
- "11211:11211"
database:
extends: meta
image: mysql:5.6
container_name: digital1st-backend-db
ports:
- "3306:3306"
volumes:
- mysql-data:/var/lib/mysql
- ./docker/docker.cnf:/etc/mysql/conf.d/docker.cnf
volumes:
mysql-data:
networks:
local:
#INSTALL-PROJECT-ON-OSX
version: '2'
services:
app:
volumes:
- drupal-vendor:/drupal/app/vendor
- drupal-modules-contrib:/drupal/app/docroot/modules/contrib
- drupal-themes-contrib:/drupal/app/docroot/themes/contrib
- drupal-profiles-contrib:/drupal/app/docroot/profiles/contrib
- drupal-core:/drupal/app/docroot/core
volumes:
drupal-vendor:
drupal-core:
drupal-modules-contrib:
drupal-themes-contrib:
drupal-profiles-contrib:
#RUN-PROJECT-ON-OSX
version: '2'
services:
app:
volumes:
- digital1st-backend-app-sync:/drupal/app:nocopy
- drupal-vendor:/drupal/app/vendor
- drupal-modules-contrib:/drupal/app/docroot/modules/contrib
- drupal-themes-contrib:/drupal/app/docroot/themes/contrib
- drupal-profiles-contrib:/drupal/app/docroot/profiles/contrib
- drupal-core:/drupal/app/docroot/core
volumes:
drupal-vendor:
drupal-core:
drupal-modules-contrib:
drupal-themes-contrib:
drupal-profiles-contrib:
digital1st-backend-app-sync:
external: true

The #BASE is the project, it is used whenever you work on the project.

The #INSTALL-PROJECT-ON-OSX is used only for the first time you set up the project. It was made this way because there is a conflict between Composer and docker-sync volume, which raise an exception when Composer tries to install dependence. I didn't figure out how to solve it.

The #RUN-PROJECT-ON-OSX is used to run the project, after you've setup the project.

In the #INSTALL-PROJECT-ON-OSX and #RUN-PROJECT-ON-OSX I created specific volumes for Drupal folders. It is important for docker-sync because avoid syncing these folders, which make docker-sync startup very fast.

Without these volumes, docker-sync startup takes almost ~5 minutes. With these volumes, docker-sync takes ~35 seconds. It is a really good improvement.

5 — Makefile

SHELL := /bin/bash # Use bash syntax
.PHONY: run in mysql stop clean build storybook
UNAME_S := $(shell uname -s)
# Configure environment.
# ----------------------
export TZ=America/Sao_Paulo
export USER_ID=$(shell id -u)
export GROUP_ID=$(shell if [ `id -g` == '20' ]; then echo '1000'; else echo `id -g`; fi)
install:
@if [ $(UNAME_S) = "Darwin" ]; then\
docker-compose -f docker-compose.yml -f docker-compose-install-osx.yml run --service-ports --name digital1st-backend-app --rm app;\
else\
docker-compose -f docker-compose.yml run --service-ports --name digital1st-backend-app --rm app;\
fi;
run:
@if [ $(UNAME_S) = "Darwin" ]; then\
$(MAKE) delete-unsync-files;\
docker-sync start;\
docker-compose -f docker-compose.yml -f docker-compose-run-osx.yml run --service-ports --name digital1st-backend-app --rm app;\
else\
docker-compose -f docker-compose.yml run --service-ports --name digital1st-backend-app --rm app;\
fi;
down:
@if [ $(UNAME_S) = "Darwin" ]; then\
docker-sync clean;\
fi;
docker-compose down
destroy:
docker-compose down
docker rmi digital1st-backend-app
docker volume rm digital1st-backend_mysql-data
rm -f docroot/sites/default/settings.local.php
rm -Rf docroot/sites/default/files
$(MAKE) delete-unsync-files
@if  [ $(UNAME_S) = "Darwin" ]; then\
docker-sync clean;\
docker volume rm digital1st-backend_drupal-core;\
docker volume rm digital1st-backend_drupal-vendor;\
docker volume rm digital1st-backend_drupal-modules-contrib;\
docker volume rm digital1st-backend_drupal-themes-contrib;\
docker volume rm digital1st-backend_drupal-profiles-contrib;\
fi;
delete-unsync-files:
rm -Rf docroot/core/*
rm -Rf vendor/*
rm -Rf docroot/profiles/contrib/*
rm -Rf docroot/themes/contrib/*
rm -Rf docroot/modules/contrib/*

As I've explained, I created a docker-compose to install and another to run, so I also create a run command and install command in the Makefile. Notice the command docker-sync start is only execute on run command because of composer conflict I've said before.

To install the project, open a terminal session, go to the project folder and type make install . If all goes to plan, you will see in the terminal:

--------------------------------------
--- SUCCESS!! \o/ ---
--------------------------------------

After that, you can type make run . This command will keep a container session opened ( drupal@app:~/app$ ). It is useful for run Drupal commands such as drush cr , composer install , etc.

In the next time, you will work on the project, you only need to type make run .

Whenever when you finish your work, you can type the command exit inside the container session and, after session close, type the command make down to stop docker-sync and remove containers.

The command make destroy should be executed only when you wont work on the project anymore.

Conclusion

I don’t know if this is a good or bad approach, but it saves a lot of time in my development process daily basis