How to build multiple web apps with Elixir thanks to umbrella — part 2: Set up the project

A (wannabe) guide to learn how to create web apps with Elixir 1.9, Phoenix 1.4, Docker (for development), Git (including submodules) to version each app, PostgresQL, Redis, API calls and deployment.

This guide is addressed to developers who have a few projects behind them and are willing to give a try to Elixir. This second part will tell you how to set up a project.

Summary

• Why Elixir?
• Set up the project
• Create users for the admin website part 1: Bootstrap, entity and tests
• Create users for the admin website part 2: security and passwords
• Deploy the Umbrella App
• Create an API
• Create an aggregation service
• Handle channel for real time applications

New! You can now find the guide sources here: https://github.com/aridjar-umbrella-guide/medium_umbrella! I’ll upgrade them at the same time I upgrade the guide!

In this chapter, we will see how to install, create and configure an umbrella app and its children app, docker and save each project in a git repository.

For this guide, it is advised to know how to create a git repository as we will just explore the submodule part of git in the associated part. You won’t need any knowledge on docker.

Install Elixir, Docker and Node

For those of you who don’t know docker, we could create a Dockerfile and use it to do every installation and command linked to the language from a controlled virtual environment. But it would force everyone to use docker, and some of you may want to avoid that.

So, first of all, let’s install Elixir by following this guide: Installing Elixir. Once it is done, open a terminal and type

$> mix -v

You should have a similar output:

Note: you may want to check out the Erland/OTP version in docker to be sure to have a matching version (otherwise, it can create a problem). To do so, go in the docker-elixir repository, select the latest version (actually 1.9) and open the file named Dockerfile. The first line should tell you the used version. If you version is different, please install Erlang.

Once Elixir is installed, let’s take care of Phoenix. As it is Elixir’s main framework we will use it to make our web apps.

$> mix archive.install hex phx_new 1.4.2

If this doesn’t work, you should be able to install it through Github with the old command:

$> mix archive.install https://github.com/phoenixframework/archives/raw/master/phx_new.ez

Then, let’s install Docker.

Note: it is not mandatory, but I may not provide other ways to configure and handle different things, such as the databases, through this guide.

Once Docker is installed, restart the shell and enter:

docker -v

You should see the version displayed. If not, add the Docker folder to your $path.

Lastly, if you don’t already have Node installed on your computer and you want to have it, follow this guide: install node.js . It is however not mandatory, as we will compile and run node and the assets in a Docker container.

Create the umbrella app

Now that we have installed everything we need, let’s talk a little bit about umbrella applications:

An Elixir Umbrella application defines one parent application that houses and manages a set of child applications.

This means, instead of creating multiple apps separated from each others, each app will be linked to the others through the umbrella, allowing them to have both the advantages of micro-services and monolith applications.

But enough talk! To create the umbrella app, you’ll have to enter:

$> mix phx.new [destination_folder] --umbrella

Let’s have a detailed look at each word:

• mix is the elixir’s package manager, like NPM is for JavaScript.
• phx is the keyword linked to the Phoenix framework.
• new create a new app. If you want to know more about it, just write mix phx.new in the command shell: it’ll display the documentation. New without phx. in front create a simple Elixir app, as we will see later.
• [destination_folder] is the place you want your application to be.

Note: if it is an umbrella app, it’ll create the folder and add _umbrella after the name.

• --umbrella tell mix to create an umbrella app.

At some point, this command will ask you if you want fetch and install the dependencies. If you intend to use docker, it is not mandatory. Otherwise, type Y and press enter.

This command creates an application with two folders (apps and config) and few files, including the configuration files for multiple environments. Most of them are commons (.gitignore, README.md…).

The two files you may not know are the .formatter.exs and mix.exs. Each of them are an Elixir Script file(.exs). mix.exs handle the mix configuration for this project, include the description, the dependencies, and sometimes other things. We will talk about the .formatter.exs in a few paragraphs.

Once the command did everything it had to do, do not follow what elixir advises you to do. Instead do:

$> cd [destination_folder]_umbrella && rm -rf ./apps/*

Doing so will erase the two apps Phoenix generated for us when we created the umbrella app. Depending of what you want, you may keep them. But, as for this guide, we won’t use them: the “web” if for a public application. I use Angular on a Node server, and you probably also prefer something built for the front end, like React or VueJS.

We also want to adapt the name of the applications, thit is why we also removed the back end one.

Note: Some of you may argue that we could host and compile a JS front end project in an umbrella app. And you are probably right, but I don’t want to spend time to understand how to do it, at least for now. But if you know how to do it, write a medium article, send me a link to it, and I’ll post it here! :)

Finally, the .formatter.exs is a file allowing you to use themix format command. The purpose is to format each files containing elixir code, so they follow the same writing rules, making them easier to read for everyone.

II shouldn’t impact your code a lot, as functional languages aren’t as permissive as iterative or object languages, especially when we follow the main idea of the functional paradigm (lots of short, pure functions), but it can be handy depending of the situation.

We will now create 5 projects inside.

To do so, let’s move to the apps folder:

$> cd apps

Warning !! Be careful: the name of each folder we will create will have to be the same as the git repository’s name. This is to avoid complications and excessive, useless configuration.

This folder is designed to contain each app we will create. The first one will be the administration website. To do so, just do the following command:

$> mix phx.new admin --app admin --no-ecto

The --app [name] option allows us to define the name of the OTP application, which is an Erlang runtime system (to oversimplify the definition: it is an independent app and a full library at the same time).

The --no-ecto option removes the steps to create the Ecto files, which is the way to interact with our PostgreSQL database.

We could use the --module option if we wanted a specific name for the main module. As we don’t use it, the module will take the name of the folder (here: admin).

Note: a module always begin with an uppercase letter.

I keep the exploration of the created apps for the next guide. The only modification we will do in this part will be in each configuration files, after the docker set up.

As we put the --no-ecto option, we need to create an app to interact with the database:

$> mix new database --app database --sup

We don’t build a Phoenix app here, but just a normal Elixir app, with a supervision tree thanks to the --sup option. This option is sort of automatically done in a Phoenix app, as it is mandatory for it to run.

Here is the command to build the API app:

$> mix phx.new api --app api --no-webpack --no-html --no-ecto

The --no-webpack option removes the steps to create the CSS and JavaScript base. As we do an API we don’t need any of them.

The --no-html option cancels the step creating the templates.

Note: We will compare Database and Admin during the next guide, and Admin and API in the “Create an API” part.

Let’s create the two last apps:

$> mix new aggregator --app aggregator --sup
$> mix phx.new webhook --app webhook --no-webpack --no-html --no-ecto

I separate the API, the aggregator and the webhook app as I want them to work separately. As each app can be independent, you may want to stop your API but not the aggregator or webhook if there is, for example, a security breach.

The purpose of the Aggregator will be to get posts from your twitter news feed, and the Webhook will be here to send the data to people connected to your application. Because Aggregator shouldn’t be called (but it should call), we don’t need Phoenix here. We will use Poison to handle the HTTP calls, but we will see this in the appropriate part.

We are done here! But before going to Docker, we need to add one line in one file:

• In apps/admin/assets/package.json add , "build":"webpack --mode development" in the scripts part. Doing so will avoid some problems later.

Let’s handle Docker now

As the umbrella project and the sub projects are created, we can focus on Docker. To do so, we need to create three files:

• a Dockerfile
• a docker-compose.yml
• a .env

The Dockerfile creates an docker image, which is used in a container, itself being:

A lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.

To create a Dockerfile, open your IDE to the umbrella app’s root and add a file called Dockerfile . It will contain a list of commands. This list will be executed once fully, each commands in an individual temporary container. And so to create the full image that we will run after.

Next, we will work on the Docker-compose:

Docker-compose is a tool for defining and running multi-container Docker applications.

The docker-compose file will define which container will be used. It also define some configuration and the env variables. The docker-compose file must be created at the root and called docker-compose.yml . Once it is created, put the following code inside:

As we have include environment variable, we need to build a .env:

A good practice is to add the .env to the .gitignore and create a .env.dist to share the structure without sharing the data you put in it (protecting password, API keys, etc.).

Once the docker-compose.yml and the .env files are created, we need to build the docker image:

$> docker-compose build

Once the build is done, let’s try our docker-compose:

$> docker-compose up

You may have to share your local storage or do some specific configuration for docker. You may also experience a shut down of the elixir server, and if not, you will probably have error messages. This is normal, as we didn’t configure anything, and even if it is working, it isn’t ready yet.

Kill the process with a ctlr+c, and then just launch Postgres and Redis in silent mode with the following command:

$> docker-compose up -d postgres redis

They will run as background tasks, so we won’t have to care about them and we won’t be disturbed by their logs.

Now, let’s get the IP of the Postgres container so we may use it latter in our configuration files. To do so, enter:

$> docker ps
$> docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' [postgres_container_id]

docker ps show the list of running container. With the id of the Postgres container, use the second command to extract the IP and save it for later.

The docker inspect allow us to get data about a specific container. The passed option allows us to only get the IP of the container, which we will use in our .env file. To do so, replace the PSQL_HOST value with the result you obtain from this last command.

Note: always launch postgres first and redis second, as the IP is determined by the launch order.

To simplify our life, we will manually create the database, with an important command: docker-compose exec. To do so, enter:

$> docker-compose exec postgres psql user user

docker-compose exec execute a command in a specific container (here it is in the postgrescontainer).

psql, which is the command, is the Postgres’s shell, and it needs a database and a role to connect (here user and user). To create the dev and the test database, enter the following command:

CREATE DATABASE test_dev;
CREATE DATABASE test_test;

To check if the database are created, enter \l and press enter. At the end of the list, you should see the two databases.

To leave PSQL, just enter \q and press enter. Finally, to stop all the containers, you can use:

$> docker-compose stop

Config apps

Now that we have our 5 apps created and docker working, let’s do a little bit a configuration, first in the 4 phoenix apps:

• In each app created, but database, open the file config/devs.exs and config/test.exs
• In dev.exs, change the port to 4001, 4002, 4003 and 4004 (1 port per app)
• In test.exs, change the port to 4101, 4102, 4103 and 4104 (1 port per app)

That’s it for the phoenix apps. But we also need to configure the database app. Every modification here will be done in apps/database:

• Create the config folder. Inside, create 5 files: config.exs, dev.exs, prod.exs, prod.secret.exs and test.exs
• In each files, add use Mix.Config at the beginning
• In config.exs, add the following lines:

config :database,
  ecto_repos: [Database.Repo]
import_config "#{Mix.env()}.exs"

• In prod.exs, add the following line: import_config “prod.secret.exs"
• In each other files, add (watch out the 2 last lines):

config :database, Database.Repo,
  username: System.get_env("POSTGRES_USER"),
  password: System.get_env("POSTGRES_PASSWORD"),
  database: System.get_env("POSTGRES_DB_ENV"), # replace env with dev, test or remove it depending of the file
  hostname: System.get_env("POSTGRES_HOST"),
  pool_size: 10 # in dev.exs et prod.secret.exs
  pool: Ecto.Adapters.SQL.Sandbox # in test.exs

Finaly, let’s add Ecto to the Database app:

• Open database/mix.exs, in the project list, add aliases: aliases().
• Still in database/mix.exs, in the deps list, add {:ecto_sql, "~>3.0"}, and {:postgrex, ">=0.0.0"}.
• Once again, in database/mix.exs, add the aliases function such as:

defp aliases do
  [
    “ecto.setup”: [“ecto.create”, “ecto.migrate”, “run priv/repo/seeds.exs”],
    “ecto.reset”: [“ecto.drop”, “ecto.setup”],
    test: [“ecto.create — quiet”, “ecto.migrate”, “test”]
  ]
end

• Now, open lib/database/application.ex and add Database.Repo in the children list
• Finaly, create lib/database/repo.ex and put inside the following:

defmodule Database.Repo do
  use Ecto.Repo,
    otp_app: :database,
    adapter: Ecto.Adapters.Postgres
end

Now you can breath, the configuration and the set up are over.

With the configuration done, let’s try again our Elixir container:

$> docker-compose up -d postgres
$> docker-compose up elixir

Go to your browser and type localhost:4001 . You should see something like this picture, meaning Phoenix and Docker are well configured:

Let add a touch of git submodule

Now that we have our project ready, let’s create a git repository for the umbrella app.

Important: as the umbrella contains other projects, don’t add, commit and push anything for now. Just link your project with a git repository (git init and git remote add origin).

As this guide is addressed to people with some git experience, I expect you to know how to do so. If not, follow theses guides:

• If you create an open source project: Github
• If you want a private repo: Gitlab

In the case you create a open source Github project, don’t add a .gitignore during the git project creation. You can still create a LICENSE file. If you do so, before pushing, you will have to do:

$> git pull # it should fail, but allow you to have access to master
$> git --set-upstream-to=origin/master master
$> git pull --allow-unrelated-histories

Once your project is created and added to your repository, let’s move to the apps folder.

$> cd ./apps

Git submodule

Create a new project on git for each app, with the same name as the app folder. Once again, this is extremely important to avoid having to do a lot more configuration than necessary.

Link each app to the associated git repository and push them. Once it’s done, go back to the apps folder. From here, you just have to enter 5 time the same command:

$> git submodules add [repository_url]

Once the first command is done, a new file will be created: .gitmodule . It contains the list of each submodules used.

A word about the submodules: you should be able to update them at the root of the project with a simple git pull . But, like me, you may also need to pull in each apps. I’m probably not as good as you may be with them.

The idea of using the submodules is to allow other people to access only the selected apps you want them to access, but also to differentiate any of your apps, as they may not need others app to work. This make you also able to keep a separated track of the evolution (without every branchs), etc.

Important: You can now add everything to the git repository of the umbrella app, commit and push.

To finish this guide, we can do two non mandatory commands:

$> mix deps.get

Which download the needed Elixir’s dependencies.

$> cd apps/admin/assets
$> npm install
$> npm run build

This download the JS dependencies and compile them (and the CSS) in the /priv/static. Warning: by doing so, you may overwrite the static in the docker container as we didn’t put the path of this folder in the docker-compose.

That’s all for today! I hope this second tutorial pleased you and helped you to set up an umbrella Elixir project.

Next guide, we will create the user part of the administration website, and add some bootstrap to it.

Until next time, happy coding!

Edit 18/03/2019:

• Change all docker files (including the advice from Eduard Liberumed about the env settings)
• Add a fifth app to handle all interaction with the database, more in line with the idea of a security breach from APIs (with the initial idea, stopping API would create errors in others app is it was API which was supposed to handle the datas)
• Remove Ecto from the four first apps
• Add Ecto setup to the fifth app as it isn’t a phoenix app

It took me time to come back to writing, and I apologize for that to any one who wanted to learn more about umbrella project through this guide. Life just happened.

Edit 19/03/2019: add few details in the git part

Edit 20/03/2019: add a line in the Dockerfile RUN apt-get install -y inotify-tools

Edit 03/08/2019: update to Elixir 1.9

Edit 17/09/2019: change Aggregator from a Phoenix application to an normal application.