Build once, deploy everywhere — part 1
In the DevOps community, there is common agreement on avoiding environment-specific builds. One aims to build the deployment package once, for any environment, and set configuration data at deploy time. Such configuration data can be stored in one or more locations; typical examples are launch parameters, environment variables, configuration files, distributed services like etcd, ZooKeeper or Consul.
That’s the ideal scenario, but in reality some frameworks make building a single package for all environments harder than it should be.
The challenges
The “only build your binaries once” principle is not new: some traces goes back to the ’90s. It came into spotlight with Humble & Farley’s Continuous Delivery book [1]; chapter 5 states it explicitly but it permeates every page.
One might think there is nothing wrong with rebuilding from the same source: just pick a commit ID (or tag or label — whatever your version control uses to select a version), reset the build workspace to the version of the code matching the ID, et voilà, you rebuild the exact binaries from the same source. Or not?
Reality is different from theory due to a simple concept:
the build is not just your code
Consider what’s needed to run a build:
- Operating systems (at build and run-time)
- Shared libraries and frameworks (Java JDK, .Net Framework, etc.)
- Build toolchain (compiler, linker, minifier, package manager, transpilator, etc.)
- Packaged libraries
- Your code
While it is conceivable to save every possible element that comprises your build environment in version control, it is impractical to recreate the build machine from scratch every time. It requires a lot of time and resources.
So, which element is the most fragile? Where to best invest your limited time and resources to control the reproducibility of your build?
Library dependencies
While the answer may be different in some unusual scenarios, I think anyone will agree that the most fragile layer in a build stack is the set of libraries the application code depends on.
Managing dependent libraries is so important in modern development, that any language platform of the last 20 years has some de-facto standard package manager in its ecosystem to harness this problem — npm, Yarn, Maven, NuGet, pip, gem etc. More recent languages even come with a “built-in” package manager (dep for Go, Cargo for Rust, Swift Package Manager).
If the development team is not strict about dependencies, running the same build script twice from the same source code may produce different results: more recent versions might be pulled in, build might suddenly break due to conflicting dependencies, Semantic Versioning rules not respected. Some toolchains also uniquely tag binaries (e.g. time-stamping) so it is virtually impossible to get identical files.
One can do a stricter grip on dependent libraries via two mechanisms:
- State the exact library version to use (often called pinning)
- Pull libraries from a controlled location, such as a network share or a full-blown artefact server (Nexus, Artifactory, etc.)
In practice, pinning specific versions for libraries in use can be difficult, and even counterproductive: libraries are regularly updated and when an update includes security fixes, you have the moral responsibility to update.
Testing limitations
What is the consequence of this inability to reproduce the exact same artefacts from sources? One should consider such artefacts as a new version of the system, which means that tests must be re‑executed.
People have experienced all kind of run‑time issues as a consequence of an updated library: code breaking on some operating system versions, visual bugs unnoticed by automated testing, etc. Most importantly, you will not be able to isolate differences between environments to the configuration. One spends a lot of time tracking the cause, while banging one’s head because code hasn’t changed.
What could possibly go wrong?
Given the above considerations, I am still amazed to discover recent developer platforms requiring one to build artefacts for a specific environment configuration (dev, Q&A, prod). The JavaScript community seems to lean on this approach more than others.
Why they are doing this? Because the configuration data is embedded in code, so artefacts hold configuration data for a specific environment. To target another environment you need different artefacts, i.e. another build. Because of the different artefacts, integration or end-to-end tests will test one version, but can potentially give a different outcome when run on a different environment.
I insist that rebuilding the code invalidates testing, but there is an additional reason to avoid embedding configuration data in code.
Many configuration items control how the system behaves in production. The ability to change them at the snap of a finger is critical for performance and resilience. As an example, a system administrator sees an increase in load on the database and client timing out: she can change some connection parameters and temporarily raise the timeouts (I am not suggesting that tweaking timeouts is a long term solution, but a possible action to gain time in a critical scenario). Should she wait 15 minutes for the npm build to complete during Black Friday? Another example is DNS management: system refactoring, company mergers and acquisitions, hacker attacks, marketing campaigns, change of load profiles are all good reason to change, temporarily or permanently, DNS names.
We can summarise in a short line:
developers do not own configuration data
Configuration data includes server names, database names, users and passwords, secrets, external services, timeouts, retry policies, URLs and more (this is a huge topic and we suggest the reading of Release It! [2]). All of this data must live outside your code, easily accessible to an administrator: as a developer, you should be aware if your toolchain does this for you.
Wrapping up
In this first part, we went through some general issues regarding build fragility and configuration data lock‑in, stressing why you should deploy the same artefacts on all environments. In the next instalment we will see some real scenarios and how to solve them.
References
1. Jez Humble, David Farley — Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation — Addison-Wesley 2010
2. Michael T. Nygard — Release It! Design and Deploy Production-Ready Software — O′Reilly 2018, 2nd ed.
