Nov 11, 2020
Welcome Aboard the Pipedrive iOS Release Train
At Pipedrive, our web app has been able to manage deployments of new changes up to 500 times per week, this sounds pretty unrealistic in the iOS world. With each build requiring app store reviews, Apple pushes towards not only rare, but feature-rich releases.
4 to 5 years ago, when Pipedrive mobile apps were still young, we followed the approach of releasing once a month or sometimes less often. At some point, with Pipedrive gaining momentum and the customer base growing, this system became a blocker for maintaining quality releases, fast iteration, and experimentation. With more than 17 languages supported, growing app complexity, and new features popping up faster and faster, it became increasingly clear that the default approach that Apple suggests — wasn’t working for us.
The first step towards improving this was introducing a 2-week release cycle. Soon after this approach we then switched to weekly releases. This required from us a lot to be both changed and improved — the details of which we are itching to share it with you. Now, fasten your seatbelts and step aboard the weekly release train of the Pipedrive iOS app!
Automation mindset
“Automation mindset” is one of our mobile tribe core values, which as a team, we have defined for ourselves.
It’s essentially impossible to maintain a weekly release cycle while also keeping development and the speed of bug fixes on par. This is especially true if time is spent on daily routines and the more boring manual work. With that in mind, we try to offload these tasks and automate as much as possible.
We automate pull request validation, PR build assembly, master and release candidate build preparation and distribution, nightly builds, localization upload and download, dependency updates, version bump after a successful release, test execution, and much more. We even managed to automate more cumbersome and disliked (by the majority of iOS developers) processes like certificate and provisioning profile management, including adding new test devices and onboarding new team members.
Identifying potentially automatable tasks, and integrating them in our daily process has been fun. Besides enjoyment, it’s also a way for everyone to learn something new by acquiring scripting skills and diving deeper into our weekly release train machine. By now, everyone contributes through one way or another to automation improvements (from juniors to seniors), and there is always something to improve!
Jenkins
Pipedrive uses a vast Jenkins infrastructure that is maintained by our incredible Tooling team. This sounds really cool… until you realize that all of the countless jobs, scripts, pipelines, and daily routines that are tuned to making the Pipedrive web app happen, are not suited to the mobile world.
That’s why, in the mobile team, everyone is also a part-time automation tooling engineer! We maintain our own nodes (5 Mac machines currently), which are working day and night to serve iOS, Android, and React Native teams’ needs.
While it may sound frightening, it’s actually not at all, especially when part of the work required for maintenance is, guess what? … Automated! New node setups in case we have to scale and node machine updates don’t require too much effort since they can be done by triggering corresponding jobs or even just through running a particular script.
Overall, the whole setup is stable and requires a relatively small effort to maintain. Yes, sometimes builds may fail and now and then some tinkering is required, but overall all 20+ iOS Jenkins jobs are up and running, letting us concentrate on what really matters.
We also created our own solution that allows us to generalize the standard work required for each job (required repo download and pre-execution setup), and keep all job-specific execution steps inside our mobile app GitHub repositories. This ensures that it’s extremely easy to spawn a new job — all that is needed is to copy an already existing Jenkins job that has only one step — triggering job script, and then adding a new pipeline script to the app repository. It’s a setup that allows for easy job maintenance and review, as well as easy job transfer if needed. One day, I may write a separate post about this setup, but overall we are really pleased with this approach and we had a decent amount of fun making it happen!
Fastlane
Fastlane, initially created by Felix Krause and now owned by Google, is widely used and loved by iOS and other mobile teams. It’s hard to estimate the amount of time it has saved us for routine around-development tasks, from app builds and test executions, to dSYM upload and hotfix routines for live builds. Every iOS release usually requires a lot of manual work, like creating a new version, updating app metadata (for all languages), manually selecting the correct build to release, etc., and Fastlane allows to automate it easily. Each Jenkins job usually just executes the corresponding action that is stored in our repo.
Since we rely on more than just the default actions provided by Fastlane, we also wrote a lot of custom Ruby code to supercharge our setup. I believe Ruby is one of the best scripting languages and is a joy to both learn and use.
A main branch that is always ready to be released
Stable weekly releases are only possible when the main branch is continuously kept in a releasable state. This is the state we try to maintain every time, and it yields the desired results. Each pull request triggers a set of jobs that check code style, run all the tests, and create adhoc build from the branch from new changes. We don’t merge it until all checks pass and a PR is reviewed by at least one team member. In case changes can potentially break something, a PR build has to be checked as part of the review. When merged, the new build is triggered from the main branch right away, running all tests again and also creating a new Alpha build.
The overall rule is that developers are the ones responsible for the quality of the delivered solutions, not QA or someone else. The aim is thus to deliver changes in small chunks, make sure that the main branch is stable at all moments, and at the same time merge new code as soon as possible.
As you might have already guessed, we strongly believe in a trunk-based development and apply it to all mobile teams. No long stale feature branches, no painful conflict merging, no suddenly broken main branch on the release day!
Fast and clear reviews
Iterating speed over pull requests also required additional improvements to the process.
We use Danger to improve our review process. When a PR is created, we run a series of code-related checks:
- The changed line count is less than 500? The PR won’t be blocked if there are more, since sometimes it’s really not possible, but the overall target is to make each PR small and clean for easier understanding and review.
- Are newly added files tested? It’s a non-blocking check, but seeing a notice that your new fancy class X is not covered by tests actually motivates you to remember to cover new code with tests. Also, you don’t want your colleagues to see this warning while reviewing your PR, right?
- If a new feature flag is added, it helps remind you that it also has to be added to the feature flag board and Firebase, since it’s easy to forget.
- Also some other small checks for fun and giggles
Danger is an extendible and flexible tool, and whenever we have an idea how to improve something in the code review process, we are eager to do it (especially since it’s possible to implement Danger scripts on Ruby, and as you remember, we looove Ruby)
We also use Swiftlint and Swiftformat that take care of almost all code formatting issues in new code. The majority of the formatting is happening via git hooks before PR is actually submitted, and in case formatter missed something, the linter check is executed as part of a regular PR validation process. This approach fully eliminates code reviews based on code style — if you don’t like specific styling, just add a new linter/formatter rule and submit it for team review! This saves a lot of time and allows us to focus on reviewing actual changes, not braces and spacings.
Translations
We use Crowdin for Pipedrive localization, and support more than 17 languages, adding new ones every year (you can read more about the overall process here). In this area, mobile apps are keeping up with the web app, so work with new translation strings is part of our weekly tasks.
Luckily, the majority of the work is automated — once per week we download newly added strings to Crowdin and upload new translations nightly. Automated PR is created and sent for review. We could have made it a fully automated process, but we wanted to add an additional verification level to the flow. A weekly release schedule and regular localization updates mean that it’s easy to roll out new features on a weekly basis and per-language.
Feature flagging
This section brings us to a very important part of our process, a process that allows us to sleep peacefully after each release — feature-flagging. We use Firebase for this and it suits our needs perfectly since it allows us to enable specific flags per language, per country, per build, or to a specific percentage of users, and also allows for creating custom criteria when needed.
The first rule that we try to follow is that — all risky changes, as well as all new features, are flagged. When a new release is rolled out and we are sure that it is stable, we start enabling them gradually. Here, processes and approaches may vary. For new features, the first rollout wave usually only includes English app version users, since there is a high chance that other languages are not translated yet. Since we iterate each week and new translations are added on a regular basis, each week we can increase the userbase that gets access to new features.
Gradual feature rollout means that it’s possible to fix bugs before the feature reaches a wider audience as well as be able to turn it off completely if something goes horribly wrong (and that’s why we can sleep peacefully after each release — there is an extremely low probability that something will break as the release is rolling out — we have full control over the process). Also, the weekly release cycle means that the amount of actual changes is relatively small anyway, so it’s easy to identify what exactly went wrong in case anything happens.
Quality
As mentioned before, maintaining high quality is one of our core values, and every team member is responsible for keeping their solutions in maintainable and stable shape. In the iOS team, this is ensured by covering new code with tests, new views with snapshot tests , and running them whenever possible.
We also have a small but awesome team of QA engineers that do manual regression for some core components while also focusing on automating their work when possible. There are extremely rare cases when a decision is made to postpone a release until next week, to ensure that we meet our quality standards. Since the release cycle is so short, in most cases it doesn’t hurt anyone — the amount of changes is not that big anyway.
Slack workflow
While the biggest part of our process is automated, it still requires manual input from time to time— e.g. QA and developers’ call to decide if we proceed with the next release and pull the trigger. To make sure it’s all happening on time and that nothing is missed, we have Slack Workflow configured. This gives better visibility to what is finished or what is still pending, and ensures weekly pulse stability. Currently, our iOS team consists of only 5 people, but it may still be hard to coordinate and ensure that we are not working on the same processes at the same time.
Release notes
As a little treat for reading this article, let me tell you a short story.
Apple guidelines are not written for everyone. Period.
Unfortunately, huge companies are often allowed to break rules more easily, while smaller ones tend to be punished for missing only tiny details.
The issue we had awhile ago that blocked us from switching to weekly releases were release notes. It’s close to impossible to generate unique release notes in a 1 week period and get them translated to all languages. Because of this, we used generic release notes that were reused every week. Soon, this caused our releases to start to get rejections — Apple required us to have unique release notes that actually describe what had changed, and that didn’t repeat previous release notes. What a bummer!
As an example, Apple is inclined to ignore these similar less important details from larger companies — Facebook has a release cycle that is even shorter than a week, but they kept their release notes similar for ages. Thanks, Apple!
We found a solution by creating the so-called LEGO release note process. We designed a template that contains a set of headers, footers, and a list of feature descriptions. They all are translated and new ones are added when needed. Time after time, we trigger a Jenkins job, select a set of features from the list, and get a PR containing newly generated release notes that are unique and already translated. Once merged to the iOS repo, Fastlane takes care of uploading them and applying them to the upcoming release. Thanks to Fastlane, it’s also easy to add custom release notes when needed, e.g. when new major feature is released.
Final stop
We’ve reached our final destination, I hope this was a fun ride and you were able to learn something interesting and valuable from the journey. Now it’s time to hop off since we have to prepare for the next ride.
(You know, because they happen weekly and we are proud of it!)
