Running Madoc Platform using Docker

The Madoc platform is currently made up of 4 services with the upcoming search features adding 3 more services. For us, it’s been important to have the entire Madoc Platform up and running quickly both locally and on a server.

You will need an up-to-date version of Docker, PHP and git source control with some basic knowledge of the command-line to follow along with this article.

Local Madoc Platform

A local environment is perfect for investigating if the Madoc Platform will work for your institution or perhaps for yourself to organise some IIIF for personal use and display. This is also the easiest way to develop custom themes and modules for Madoc, enabling you to see changes as you make them.

To get started, you can clone the example configuration from GitHub.

git clone https://github.com/digirati-co-uk/madoc-example-config.git

Tip: You can use GitHub's new template repository feature to create your own version of the Madoc example configuration.

This shouldn’t take long, once this is completed you can cd into the folder

cd madoc-example-config

and run the following docker command

docker-compose up -d

You should now have a running Madoc Platform locally running at http://localhost:8988/ which you can login to.

By default you can log in with the following details:

username: admin@example.org
password: Testpass123_

The first thing we need to do is change these. Let’s create a new admin user using the example configuration. First, let’s bring down our containers.

docker-compose down

Docker compose down command will stop and remove our containers, so any data in them that is not volumed will be lost. This is useful for local environments where you want a sandboxed environment.

Tip: Check the end of this article for details on how to set up volumes and have some persistence when you restart your local stack.

Creating a new user

Now we’ve got our containers stopped again, we can start to create an initial database to seed our Madoc instance. Let’s use one of the build-in tools to create a new user.

> bin/generate-user
usage:    bin/generate-user <username> <email> <password> [group]    
example:    bin/generate-user username email@example.org pass123

Running bin/generate-user shows us the help text, and how to use this command.

Tip: you may have to run php bin/generate-user on some OS’s to get this script to work.

Now let’s create a user.

> bin/generate-user testuser test@example.org pass123
LOCK TABLES `user` WRITE;
/*!40000 ALTER TABLE `user` DISABLE KEYS */;
INSERT INTO `user` (email, name, created, modified, password_hash, role, is_active) VALUES ('test@example.org', 'testuser', '2019-06-03 12:52:39', '2019-06-03 12:52:39', '$2y$10$7T46YuFFzq3Vzf5duLYjHO4uWmvomRoR0ECY0Zk1wQD3I/8r8CY8q', 'global_admin', 1);
/*!40000 ALTER TABLE `user` ENABLE KEYS */;
UNLOCK TABLES;

The tool creates raw SQL. When you start up a fresh database in the madoc example config, it will take all of the SQL inside the ./database/sql folder and add that to your starting database. You could, for example, grab a full database dump from a deployed Madoc and chuck it in there to get a recreated environment to debug or test. However we’re going to use it to configure the default users.

Madoc has a set of default SQL that gets merged with anything you put in that folder. You can see all of the files here. If you create files that match the names in there, they will replace the originals, so be sure to copy them first and modify with your additional content, or just create a new SQL file.

What we want to do is secure our install, and remove the default admin user. Let’s create a file that matches the one inside the Madoc repository. (This file does already exist in the example config).

touch ./database/sql/09-user.sql

Now you can paste in the output from the bin/generate-user command above. Since we don’t want the test user we can replace all of the contents of that file.

You can combine the two steps we just did. If you want to override the current admin you can do something like:

$ bin/generate-user user test@example.org password > ./database/sql/09-user.sql

Tip: If you want to add another user, you can simply replace the > with >> and it will add it to the end of the file.

The example configuration is not too big, so it’s small enough to read through the source and see what’s going on outside of the theme. You will notice references to example names, which you can go through and change to whatever you like.

Theme customisations

We’ve now created a new user, hopefully with a more memorable login details. You can now run your Madoc instance locally again. Check out http://localhost:8898/s/default/page/homepage to see the default them that comes with the configuration. Everything loaded here is in your Git repository that you cloned.

CSS

The CSS for your theme can be found at custom-madoc-theme/asset/css/style.css and contains a reset for the UI contained inside Madoc. The CSS isn’t very specific and should be easy to override. From here you can start to make changes, reload your local Madoc and see them appear. Most of the customisations and branding you will need to do can be done from the CSS.

You don’t have to stick with CSS, you can write Sass or Less and have them output to the style.css and use that instead. Madoc doesn’t force any FE tooling or process on the theme. In a future article we will cover creating a theme using Webpack and Babel.

JavaScript

The default theme implementation contains no JavaScript. If you want to add JavaScript to your theme you can add a line to your layout to include it. Have a look at the Layout template to see where new JS and CSS files can be added. These are paths relative to the theme or from CDNs.

Templates

We just took a look the layout template, and you’ll notice that this is a PHP-HTML template. This is what Omeka uses for it’s templates, and you may find that some customisations you make will require you to create .phtml files, but for the Madoc templates, we went with Twig.

Overriding templates is fairly straight forward, if you know what you’re overriding. In each theme there is a folder called view that contains folder and files. Each path matches an existing template in Omeka or in Omeka modules. If you create a template with the same name, in the same path, this template will be used instead of the default one from Omeka or the module. The common pattern is to take the existing template, copy it into your theme and start making changes to it.

You can grab a snapshot of all of the custom theme files from the GitHub releases page under theme.zip which contains all of the possible templates you can override.

If you go into, for example, Annotation Studio module you’ll see a structure similar to your theme, with a view folder. If you wanted to change how the Deep zoom viewer looked, you could copy ./view/annotation-studio/viewer.twig into your theme at the same place, and start changing the markup, adding wrappers or whatever styling/classes you need.

In the future Madoc documentation will contain a list of these templates and what they do, so you will be able to better discover which templates you may want to edit to really customise every aspect of the Madoc Platform.

Docker volumes for local development

If you’re developing a theme locally, you will want to make sure that you keep your Docker compose volumes up to date. If you look at the example configuration you will see that both the example theme and the translations folder are volumes. When they are volumed, you can change the files and immediately see the changes running in your VM.

You can add more folders here for themes or even custom modules. Make sure that when you finish development that you addCOPY statements to your Docker file to copy in any new files or directories that you may have volumed. This will ensure all of your work is included in your built Docker image.

Deployment

Once you’ve got your customisations done, and you’re ready to deploy, you have a few options. You can build, tag and push your main Madoc docker image into somewhere like Dockerhub and deploy it like any other Docker container. However you will need to ensure that you correctly configure environment variables and also deploy the extra services required; MySQL, Elucidate and optionally Elasticsearch with the indexer and search service.

That process may be covered in another article, for this article we are going to assume that this Madoc instance will be a small-scale deployment. We will use docker-compose itself to mange the containers we build.

The simplest way to get deployed is to clone your configuration repository and follow the same steps for a local environment. You may want to create a new docker-compose for your deployment. You can use a lot more options to manage memory limits, health checks and other docker features not covered here.

In this new docker compose you will also want to add some additional volumes.

• Files — found at /srv/omeka/files will ensure any files that are uploaded to Omeka are saved in the host and not lost when you restart.
• Omeka database — /var/lib/mysql will persist all of the data in your site
• Annotation database — /var/lib/postgresql/data will persist all of your annotations.

Tip: You can see an example of a docker compose targeted for servers here.

Going further: NLW case study

The National Library of Wales recently migrated to Madoc from an older development version of the platform. They are using a Madoc configuration GitHub repository, as described here and have set up continuous integration to a staging environment whenever a new GitHub release is created. Here’s how it works:

• Every commit is picked up by Travis CI.
• Travis will build each docker image to ensure there are no errors.
• Travis will run docker-compose up to bring up the stack and run a set of smoke tests to ensure nothing has broken.
• If it was successful then the build is marked as passed.
• If the commit is on the master branch, the build images are tagged and pushed to Docker hub, tagged with master and the short commit hash.
• If the commit is then tagged with a version, travis builds it again this time creating a new tag latest and also the version specified, both pushed to Dockerhub.
• If everything succeeds on the tagged build, travis will tell Amazon ECS that there is a new docker image and it should pull it down and update the staging environment.

With this automation, it becomes easy and safe to deploy. You can check out NLW’s Madoc configuration. All of the automation is included in the example config in the .travis folder and you can find the required environment variables to set up in travis inside of those scripts.

This model can easily be expanded with more tests and fixtures to improve the safety of deployments.

Get involved

If you want to get involved, reach out on our GitHub with your use case or contact us through our website.