CLI tool software release process
Today I’m going to tell about the software release process that I use as the creator and software maintainer for the Command-Line Interface tool of Liferay’s DXP Cloud.
we is the application name of this tool I have been working. I created it using the Go programming language as, originally, WeDeploy’s client application (and now, DXP Cloud). It enables our users (software engineers) to deploy Docker containers on our upcoming Kubernetes cloud, configure resources such as domain names, access their services, watch logs, and more.
Previously, I maintained NodeGH. It is written in JavaScript for Node.js, so naturally, we used npm as NodeGH’s package manager. In Writing a modern CLI tool, I explain the reasons behind the decision to use Go for this new endeavor in detail. Two and a half years and 215 releases later, I can attest this was the right decision.
While the content below is rather Go specific, the ideas are language agnostic and may be useful for programs written in any programming languages.
Building the application
Building a Go application is straightforward, especially when you are either using the new Go modules or when you commit your vendor directories, as most Go projects do.
Just execute go build in the main package directory, and you are good to go.
However, if you are building for redistribution things get slightly more complex.
Beyond “go build”
To build for multiple platforms, you need the GOARCH and GOOS environment variables to set your platform target.
You also want to add some build information on your program, such as your version tag — and you should probably be using Semantic Versioning. Probably build time and build commit too.
You may want to trim absolute path on your generated binaries too. While they are useful for debugging, they might be confusing and perhaps a minor privacy nuisance.
Package distribution
How do you distribute a compiled program nowadays? In our case, this matter had to be divided into two parts: Installing and updating.
There are other ways, but this seems the most reasonable and efficient for meeting our needs.
I decided to use equinox.io, a web service for helping you package and distribute Go software for the following reasons:
- Cryptographically signing your releases is dead simple and means safer releases.
- Δ delta updates, thanks to binary diffs.
- Managing software releases is a no-brainer.
This web service was created by Alan Shreve (@inconshreveable) — better known for ngrok, and by Kyle Conroy.
Using go build directly, you would need something like this:
go build -ldflags="-X 'github.com/wedeploy/cli/defaults.Version=xxx'" -gcflags="-trimpath=$GOPATH" -asmflags="-trimpath=$GOPATH"
- Set the path accordingly to your root, if you are not using $GOPATH.
- defaults.Version must be a variable. If you use a constant, the replacement is going to fail silently.
Installing
In the cloud we have the container specification, on mobile we have sandboxes and users need to grant permission to applications to access their data.
Despite some advance, old school personal computers still lack fine control of user permissions, meaning we have a great responsibility not to let down users who trust us to run arbitrary code on their computers.
Anyhow, one such advance is origin verification. Operating systems like macOS and Windows nowadays typically ask the user whether they want to run a downloaded program.
I’m going to provide more information about the arduous code signing process and why it is something important in an upcoming blog post. For now, I want you to know that unfortunately there isn’t currently a way to code sign a package installer using equinox.io.
Therefore, I had to do some magic: create an auto-installer program and code sign it myself for Windows users because on Windows the approach of installing applications using curl wouldn’t work reliably on it.
There is a trade-off here: the very first time the user executes the program, it auto-updates to the latest version and reruns the command (in a forked process) passing the very same arguments, environment variables, and streams. A loader is shown saying the program is finishing the installation. It requires an Internet connection.
On macOS and Linux, using the common curl | bash alternative is the suggested way to install the tool at this time, even though drawbacks exists and moving forward to an installer is the best option on the long run.
Updating
From time to time using some heuristics, during regular use of the CLI, a new process is spawned in the background with a short life to silently check if there is a new version of the tool available. Once a new version is available, the user is going to be asked to update it.
The we update command contacts equinox and checks for newer versions. Any update must be signed cryptographically with an asymmetric key. The current version of the program uses an accompanying public key to verify the authenticity of a new available version. The associated securely stored strong private key is used to sign new builds of the software.
If a new update can’t be verified against a matching public key, no update happens.
A possible improvement is to add a revocation endpoint that can be tested for assurance that no private key was compromised.
If you allow downgrading your program (or installing non-semver versions), you should prompt the user for confirmation.
It is tempting to provide an auto-update capability but your user might feel betrayed. So if you go into this route, I highly recommend you make it crystal clear how it works and how the user can disable the feature. I erred in the side of caution.
Build server
Nothing evil shall happen: our software should be free of malware, security issues allowing remote arbitrary code execution, and so on.
This should be true not only in theory but in practice likewise. Therefore it is essential to reduce risks by using security in depth strategy.
Our CLI build server is encrypted and isolated from the rest of our build systems. Access to it is severely restricted to a need-to basis. Access is monitored and logged remotely.
Aside from backup, the code signing keys should only be stored in the build server.
Multi-factor authentication (> 2 factors) where at least one of the factors is public-key cryptography is used to protect access against unwelcome visitors.
No continuous integration is used. However, that may be considered in the future. I’m particularly worried & unsatisfied about how it might affect system security adversely.
Builds are always manually triggered, and tagged with git, and require release notes explaining the changes. For this kind of software, I believe this is the right call, and that building automatically would be a mistake.
Another mistake would be to let the build server do too much.
Testing
It is okay to run some limited testing before building but avoid anything that takes more than a few seconds or requires external dependencies. You should only release tested software, but testing is not the responsibility of the build server and should happen earlier in the process.
Functional tests are always run independently of builds. It usually is fine to skip running slow functional tests during the build phase in our case, but your mileage may vary. Don’t run functional tests in the CLI build server, though. They take too long and if you are provisioning more services than you need you may end up jeopardizing security.
Releasing a new version
- Modify code.
- Run tests locally:
static code analysis, go test (unit).
go test again, with a limited set of integration tests.
Use drone.io’s software as a substitute for Travis locally:
$ drone exec - Push changes to the code repository in GitHub.
This actually happens anytime I finish working on something and not necessary leads to a new version.
$ git push - Verify code changes passes the Travis CI.
- Changes get to the master branch eventually (right away, some times; it varies).
Enforcing branch protection is important if you collaborate with a team or if you expect users to build the program.
$ git merge feature - In case a new version has changed on the vendor directory, check if legal information was regenerated by scripts to update any modifications or check compliance.
The user can list dependencies with $ we about legal - Run functional tests.
TCL/expect tests run against the real infrastructure using test accounts. Really slow. Sometimes this is skipped. Tests are also run in a schedule on a Continuous Integration. Support on Windows is limited. Probably would be better with goexpect. - Add release note entry and tag a new version using Semantic Versioning (semver) notation MAJOR.MINOR.PATCH (i.e., 1.2.0).
$ make tag
release notes are shown on the update, also anytime with
$ we update release-notes - Push updated master branch with release notes and new tag to GitHub.
- Log into build server.
- From there, fetch tag and check signature. Checkout. Release.
$ git fetch <tag> && git verify-tag <tag> && git checkout <tag>
$ make release
Make release does a lot. It builds the application, tagging it using ldflags to mark the version, build commit, and build time. It also signs the application using public-key cryptography.
New versions are released to the unstable channel.
Only after retesting with the functional tests, manual validation, or both they are promoted to the stable channel by invoking make promote.
This process can take from minutes to days, depending on the list of changes.
Ideas for improvements
- Validate versions against downgrade by using a blacklist.
- Key revocation endpoint
- Probably a bad idea: endpoint to add a new security key (OV certificate + user confirmation would be required)
- Improve metrics gathering to understand user behavior (how often update notices are ignored, etc.)
I hope you like my approach to software releases. While I provided details for Go development, most of my process can be adapted to be used on any application regardless if they are a single binary, compiled, interpreted, etc.
Feel free to drop by and say what you think about it.
