How to automate your engineering processes
The focus of an engineer’s early career is code. We aim to code fast, elegantly and with few mistakes.
As we gain experience, we learn that code is only worth something when it becomes a working product. We also learn how much more to the job there is than coding. Internal app distribution, updating images and translations, version control, releasing to the App Store/Google Play and many other competing business priorities 🤯.
Very often, you hate the non-coding stuff. Like that visit to the gym, you find imaginative excuses to do it as rarely as possible. Like once per month. And it takes forever.
But there is another way. Build automatic processes. Let a machine do the tedious work for you. Once per week, once per day or on demand.
Can you ship your app once per week?
Two years ago we were releasing the Azimo apps once per month. We sometimes waited 30 days to release a completed feature. We then had to wait another 7–21 days until we had enough data to validate whether users liked the feature or not. If the feature failed, it failed slowly 🐢. We needed at least another month to release an iteration.
We now release once per week. This frequency brings a number of advantages:
- Fewer things can go wrong (see: Murphy’s Law). The larger release, the larger the list of things that can crash.
- Fewer things to keep in mind. Can you remember your last four weeks of code? What happens when QA catches a bug from the first few lines that you wrote? They’re unlikely to be fresh in the memory.
- The business learns faster. Less of your work is thrown in the trash because you have more chances to adapt your approach.
Identify all the steps
As junior engineers, we thought a release cycle looked like this:
a. Something wrong?
b. Fix it
How could something so simple take an entire month? Usually because important things are hidden between the lines. The reality looks more like this:
a. Manual tests
b. Write down what’s new
c. Import new assets — translations, images
d. Compile and build APK
e. Distribute APK internally
a. Something wrong? Return to step 1.
b. All fine? Continue to:
c. Build and sign production APK
d. Add release notes ✍️
e. Internal distribution of production app
f. Manual testing of production app
g. Store upload, including screenshots, what’s new, copy updates
a. Staged rollout from 0 to 100%
b. Monitor issues
c. The app is finally live to 100% of your users
How long does it take to complete the above? Those are our numbers before we improved and automated our process:
- Internal build and testing: 2–3 hours, multiplied by the number of times QA send back the build
- Testing process: 1–2 days, assuming QA is available immediately
- Production app distribution: 2–3 hours
With a weekly release cycle, that doesn’t leave much time for coding.
Why should we automate this process? Let’s consider this example:
- Run unit tests
- Run lint checks
- Build APK file
- Distribute APK internally
Done manually, each step takes me 15 minutes. To speed things up, I have shortcuts written down:
But shortcuts aren’t enough. When I run them on my computer, I can’t do anything else. I can’t code, I can’t change the repository branch. So I take a coffee break ☕️, scroll through Twitter or watch a conference video. Just like that, my focus is gone.
A continuous integration environment helps here. It allows you to run steps sequentially and frees up your machine so that when your feature branch is pushed to a remote repository, you can move on to other implementations.
Here is our CI config: Mac Pro, Jenkins, its Declarative Pipelines syntax, and configuration similar to this:
Continuous integration is like an assembly line. You set it up once and then forget it. When you get it right, every line of your code is one click away from reaching customers. Assembly, testing, distribution and many other tasks are done automatically.
What to automate?
Compiling, unit testing and distribution are boring tasks. I’m boring myself just by writing about them. But they are essential. Let a machine do the boring stuff.
But what else can be automated?
The QA testing processes:
We’ve written about this already:
“Automation frees engineers to focus on the things that matter and unleash their creativity”medium.com
Importing and updating translations
Azimo’s apps support eight languages. In the past, this meant 30–60 minutes of manual copy and pasting string resources every time we received new translations.
Today our python scripts:
- Fetch resources from our internal Translations API
- Update existing translations in
- Add new translations (we have separate file
new_strings.xmlfor new keys, that are moved into
strings.xmlwhen there are translations available)
- Build, test and lint-check app with updated translations
- Remove unused resources
Create a merge-request with all changes.
All we have to do is to click merge button. That’s it 🤓.
Build new release candidate
Our CI builds a new release candidate directly from the develop branch after just one click. The steps are:
- Update version code in
- Write the What’s new description (taken from feature branches names)
- Compile, unit test, lint check, build new .apk file
- Send .apk and what’s new .txt to Crashlytics Beta and notify all people about a new version
- Send a Slack notification that the build was completed (or not)
Build and release production app from master branch
- This automation builds, checks and pushes the app into the Play Store beta channel
At Azimo, every task that is repeatable and takes more than one minute is considered for automation. But how do we decide what to automate and what to do manually?
- Measure how long the task takes to implement
- Calculate how much time you waste doing the task manually (daily/weekly/monthly), and when the implementation will pay off
- Remind yourself how much you hate doing the task 😂
If automation of the task will pay off in less than a month, we automate it.
How to automate?
Simply by writing the code that is readable and known by as many people in the team as possible. We use Python scripts, terminal commands automated by Jenkins or Gradle plugins (external or our own). In the past, we also used Ruby and Fastlane (these are still partially used in our iOS CI).
Before picking the tools or languages for automation, make sure that writing and extending them is as simple as possible. Try to make it easy and fun, so you can focus on real challenges with all the extra time that automation brings you.
This post is a part of a series about making the software engineering process as effective as possible. For more, please read my Fail safe, not fast blog post.