Configuration Automation
We now know why configuration is important. You’ve also seen how we configure our Node.js projects. Awesome! The fruits of your labour are now starting to pay off. Now we can level-up our configuration to enable us to automate the configuration of our apps as they get deployed.
This is part three of a four-part series — you can start with parts 1 & 2 below!
We all want our applications or projects to work flawlessly. Configuration is a very important piece of that puzzle. It allows you to easily share code, and it also enables you to easily run your application in different environments.
If you have been reading along, then you will understand why configuration matters, and you will also be aware of how we configure our apps using our configuration module. In this post, we will dive in to how we configure our apps as part of our auto-deployment process.
Before we dive in there is some background information that needs to be shared about how we do things the way we do.
Our auto-deploy pipeline has three steps.
(1) The first step is to build an artefact which is stored in a central location. This artefact is tagged with the release version and is agnostic of any particular running environment.
(2) The second step is that the new version of the app is installed from the artefact that was generated in the first step.
(3) The final step is to configure the app on the server on which it is running. This post explores the third step in this process.
Each server we have runs in an environment. Production is either “green” or “blue”, and non-production can be either “dev”, or “test”. Once the new application code is deployed on the server it needs to be configured so that it is running with the right settings which match up with the environment it is running in. As a part of our development tooling for our applications we use Gulp. Within Gulp we have a task called “configure” which does the configuration for us.
An example of the configure task within a gulp file is shown below.
const gulp = require('gulp');
const fs = require('fs');
const config = require('@myob-oss/config');let runtimeConfig;gulp.task('configure:backend', (done) => {
runtimeConfig = config; runtimeConfig.express.port = process.env.EXPRESS_APP_PORT || config.express.port || null; fs.writeFile('./config/runtime.json', JSON.stringify(runtimeConfig, null, 2), done);
});gulp.task('configure', ['configure:backend']);
And the “configure” script in our package.json file looks like the following:
"configure": "rm -f ./config/runtime.json && gulp configure"The “configure” script in our package.json removes the existing runtime.json file, then runs the gulp configuration task. This effectively removes the current running configuration and prepares the app to be configured.
This is important as we run all our apps with NODE_ENV=production (even our non-production environments) as some modules have performance optimisations when you run the app in the “production” environment.
The way that the package.json configure script is executed is as follows.
NODE_ENV=green npm run configure
This tells the configuration process that we are configuring for a production green server. By removing the runtime.json file it means that the app is effectively not configured and ready for a fresh configuration. Those avid readers will remember that the configuration module works on an overlay system. It loads default.json, then overlays that with green.json based on the “green” environment.
This resulting configuration is what will be loaded into the “config” variable. In the “configure:runtime” task it then puts the “config” variable into the “runtimeConfig” variable. The configuration process then updates the express port from an environment variable. Finally it then saves the resulting config stored in “runtimeConfig” variable into the file runtime.json. Once this file is written out, the app is then restarted which it will then receive the new configuration.
In this case we are using environment variables to change the configuration. But you are not limited to using only this technique. You can also get settings from encrypted data-bags, or password-protected vaults, or from a key management server, or a secrets store, or a hardware security module.
To reduce complexity for this example, the environment variables are the clearest way to describe the process of using external configuration items to update the runtime configuration of the application.
Eagle-eyed readers will notice that the gulp task “configure” is actually a task which kicks off the “configure:backend” task. Having it set up this way will allow us to add other different configuration tasks, which will be executed when the main “configure” task is run. The next post in this series will cover the way we configure our frontend apps, so we will be adding a “configure:frontend” task.
Now that we understand the configuration process, let’s wrap it up with a bit more detail around the way we deploy our apps.
In our production and non-production server environments we use a tool called forever to ensure that our applications are always running. We’ve configured forever to watch only a deploy file, and if that file changes, then the application should be re-started. The deploy file is not in the artefact which is installed when the app is deployed (it is also ignored from our repository).
When a new version of the app is installed the code is updated. Because the forever tool is not watching these updated (or removed) files, the existing app is still running in memory. The app then gets configured using the process described above. Finally the deploy file is updated with a timestamp (so we can know when the app was updated) which causes forever to re-load the new app with the new configuration.
We have now super-charged our configuration. Our deployment process now can configure our applications for us. Configuration allows you to run the same code in different places with the only change needed to be made is the configuration settings. By including configuration in your auto-deployment process you can ensure that a newly installed or updated app will have the correct settings every time. As developers you can keep the security folks happy because you won’t be storing sensitive configuration information in your code.
It should be obvious now why configuration is important, and how it can be a powerful thing in deploying your applications.
The next post in this series will be describing the way we package and configure our various front-end applications. This will extend on the methods described in this post.
In the meantime, feel free to drop any questions or feedback into the comments section below!
