The missing handbook for Continuous Integration & Continuous Deployment (start here)

Fig 1: Software Development Workplace (taken from Unsplash)

Continuous Integration (CI) and Continuous Delivery (CD) are industry buzzwords these days for software developers. The industry is moving at a rapid pace, so it can be hard to keep up with new initiatives like the DevOps movement. This article aims to fill the gaps left by other articles in the space by speaking from a mobile developer's perspective rather than from the business/product point of view.

This article is an introductory, general article about CI and CD. I will bring you up to speed on CI and CD, their benefits and how to use them on projects. I don’t assume any previous experience and aim to make the ecosystem more digestible for beginners. While my experience is in mobile development, the benefits apply across development stacks.

Article Agenda

1. What are CI and CD?
2. Where do they fit into DevOps?
3. How do CI and CD workflows help developers?
4. Automation ⚙️
5. Configuration 📝
6. Metrics 📊
7. Where can I implement CI?
8. Final thoughts 📚

Fig 2: XKCD 1319 on Automation (taken from https://xkcd.com/1319/)

What is CI? 🤓

CI stands for Continuous Integration. According to Martin Fowler:

Continuous Integration is a software development practice where members of a team integrate their work frequently.

Continuous Integration usually means the automation of codebase builds and tests continuously during the development lifecycle, rather than deferring to integrate different features before delivery.

Continuous Integration helps development teams to identify bugs quickly, what parts work and what parts are broken. CI thus increases the visibility of project health through automation.

What is CD? 🚚

The “CD” in CI/CD can refer to continuous delivery and/or continuous deployment in different contexts. The concepts are similar and sometimes get used interchangeably. I am a fan of Martin Fowler's blog, so will be quoting him again:

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

Continuous Delivery complements CI, which is why they are often mentioned together. CD ensures that features are regularly deployed because, at any point, the current development version of the app could be deployed to production. Continuous Delivery automates the testing of code changes before they are merged into a repository, increasing the visibility of metrics with business teams. We still need to deploy the code to a production environment after Continuous Delivery.

Continuous Deployment (also CD!) is similar to Continuous Delivery, where every change goes through the deployment pipeline and is pushed into production. Continuously Delivered projects can often be changed to Continuously Deployed projects very easily, but aren’t because the company prefers following the Agile Scrum framework instead of Kanban, for example. Continuous Deployment automates the next stage of the pipeline after Continuous Delivery and reduces the workload of operations teams by automating an otherwise manual process.

CI and CD form the backbone of DevOps, which has grown so much in popularity in recent years that it’s become a movement. CI and CD have been tried and tested at scale in the real world with great results. A wide variety of companies (including Microsoft) champion CI and CD workflows because they have seen radical benefits in their own projects. We will cover the benefits of such tooling in this article.

DevOps Practices (context)

• Continuous Integration, Continuous Delivery and Continuous Deployment (CI/CD)
• Version control
• Agile software development
• Infrastructure as code
• Configuration management
• Continuous monitoring

We can incorporate CI or CD workflows within the company as a self-hosted environment, or leverage the infrastructure of large providers. Most projects use a self-hosted or hosted CI/CD provider such as Bitrise, CircleCI, TeamCity, GitHub Actions, AppCenter, or Xcode Cloud (beta).

How do CI and CD help a team? 👫

We can group the benefits of CI and CD for software development teams into 3 broad categories:

1. Automation (Convenience)
2. Configuration (Project Flexibility)
3. Metrics (Transparency)

Automation ⚙️

CI and CD are essentially forms of developer automation. CI is usually triggered by Git webhooks on changes to Pull Requests, Branches, Tags, or time so we don’t need to think about it. CI provides automation, and the main theme for automation benefits is convenience.

Time-Saving

Automation of small tasks saves lots of time in the long run through compounding. We can often save 30 minutes from running a test suite on large projects with CI, and this time compounds to 100s of hours in a year (equivalent to another 2-week release!). CI + CD can increase the capacity of a development team because every developer on the team is more productive.

Automate Testing

CI + CD automates best practices such as testing. CI + CD tests can test across a wider range of target devices (Macs, iOS device screen sizes, Apple Watches), OS versions and Development Environments (Xcode versions), making the QA process easier, and much more rigorous. Automated Unit, UI and Snapshot Testing also reduce the load on dedicated QA teams.

Integrate Stability into VCS

CI testing bakes stability into a development workflow because CI tests can catch bugs in a code review that developers may miss. We can implement a process that prevents merging PRs that fail automated test suites because CI + CD can be integrated into source control providers.

Write Better Software

Automation with CI and CD can lead to a better software team by implementing 3.5/12 steps of the Joel Test 🧪: Use Version Control, Make a build in one step, Make daily builds, and Use the best tools money can buy. CI and CD tools are generally powerful, high-quality tools, however, they are only part of a development toolkit, which is why they only tick 0.5 of that step.

Improve Developer Experience

Automation improves the developer experience because we don’t have to think as much during everyday development. While it’s always good to thoroughly test our app locally, we no longer have to worry if we forgot to.

Improve Developer Culture

Automation is a process improvement that can have knock-on effects to improve developer culture. Paying attention to seemingly unimportant processes like CI encourages the team to pay attention more generally to details. Attention to detail is crucial for building quality software, so it can encourage a culture of software craftsmanship.

Further, CI shifts blame from the commit author to the CI system for failing builds. CIs automate some of the tedious code review checks like tests, indentation, whitespace or general styling through linters and PR integration.

Rather than git blame causing arguments and friction, we can blame the process, and improve it. Ps. I would prefer if git blame was git author.

Small changes make a big difference. 🙌

Improve Developer Focus

A simple git push triggers a suite of tests automatically and also handles deployment to Testflight, AppCenter, TeamCity or the App Store. Developers need to focus on solving real engineering challenges, not manually distributing apps on AppCenter. Developers get more time delivering app features rather than manually deploying apps with a stable CD setup. The end result of stable Continuous Delivery is a higher team velocity.

Automate Complex, Precise Tasks like Code Signing (iOS)

CI automation can prevent code-signing ✍️ bugs. Managing manual code-signing configurations is a real pain point in iOS development because App Store Connect has a complex, unintuitive UX. These bugs can be almost entirely avoided by delegating code signing tasks to CI / CD. CI + CD manage the complexity of manual code-signing automatically with encoded, definite steps.

For those of you who are not iOS/macOS developers, code signing is a necessary step to deploy apps on the App Store. Code signing has lots of aspects: certificates, app IDs, provisioning profiles, and registered device lists for each target platform and target type. Example code-signing issues could include:

1. We can avoid accidentally using the wrong certificate when opening the project on a new machine. Development and Distribution certificates can be confusing for project beginners, and we may break configurations to build the app locally.
2. We can prevent creating extra provisioning profiles for an existing project. Instead of importing the missing profiles through Xcode, we may end up creating additional profiles by mistake.
3. The precious attention of more senior team members can be saved from cleaning up a cluttered developer portal after accidentally triggering automatic code-signing for a manually code-signed project.

Automation of complex steps is especially helpful for teams who are onboarding junior developers or recently joined developers but anybody can make these mistakes. Automation of complex tasks improves project stability.

NOTE: CI + CD systems are generally cross-platform so deployment configuration improvements outside the Apple ecosystem are also possible.

Notify Teams Immediately

CI + CD can also automate the delivery of notifications with Twitter, Email or Slack. By automating notifications, the productivity of a team improves because the team can respond immediately after an issue has been identified. Notifications are crucial in time-critical hotfixes but improve the response time of the team in general.

Collect App Artefacts by Default

CI + CD also automates the collection and storage of app artefacts, including IPAs, dSyms or logs. This can be convenient for debugging crash reports in production, running the app for a particular version or re-distributing failed deployments. The alternative of manually distributing or debugging apps can be time-consuming because we may not check artefacts into source control at the same time. CI + CD prevents this human error and makes artefacts more widely available (improved transparency).

Configuration 📝

CI and CD tooling need to be configured on a per-project basis. I’ve outlined the advantages of having a configurable CI and CD environment below.

Configuration as code

CI + CD can be configured as code for most tools. Committing configuration as code means we have a single source of truth for the configuration that is backed up to the cloud. This ensures stability and scalability. We prevent unexpected changes to the workflow without notifying the team more widely by restricting edit access to team leads. Version Control enables us to track changes in configuration and to roll back or diagnose breaking changes.

Furthermore, we can scale a performant, stable workflow between projects by sharing the self-documenting config. Configuration as code speeds up onboarding because developers don’t need to ask for documentation to understand a workflow if they understand the configuration files.

Customisable Notifications

Notifications can be comprehensively configured for the project needs. We can notify the team for every build, only build successes, only build failures, and similarly for tests. This makes CI and CD very helpful in multiple scenarios. Notifying only on build failures helps teams who build regularly and want to prevent spamming communication channels. Notifying every build status raises more awareness of project health. Notifications raise awareness of the project state even with non-technical team members and help to diagnose errors faster.

Optimisation of workflows

Documenting previous CI + CD workflows means that we can also compare the performance and stability of different setups. CI + CD build times for a workflow can vary depending on numerous settings, such as clean builds, target platforms and deployment methods.

The best workflow for a given project varies depending on numerous factors such as project size, impending deadlines, code coverage and dependency stacks. As such, configuration as code helps us find the optimal workflow that balances velocity, stability and maintenance.

Reduce team friction

Configuration as code reduces friction as the team grows. We can restrict access for editing workflows to team leads only, and coordinate workflows easily even as the team grows.

Cross-Platform Configuration

CI and CD are powerful, flexible tools that can usually be configured across multiple development ecosystems. GitHub Actions is a popular solution these days that can automate workflows on iOS, Android, Web and Desktop projects with one subscription. Not every CI and CD tool is cross-platform, but we can choose a solution that provides good value for money.

Metrics 📊

“What gets measured, gets done.” — Peter Drucker

Increase Visibility of Issues

CI + CD can be used to increase the visibility of key project metrics such as code coverage. Tracking project metrics increases the visibility of potential issues before they become unmanageable e.g. code coverage. Tracking project metrics helps make a more compelling case to stakeholders for refactoring or improving code coverage.

Clear Overview of System under Test (SuT)

CI saves time during feature development, that can be used to pay attention to wider project issues. Metrics like app launch times, the hang rate, memory usage, energy use, scrolling performance and networking load can be tracked through automated UI performance tests. We can quickly identify regressions, and improve them with a clearer overview of the whole System under Test (SuT). It’s very hard for bugs to hide with comprehensive metrics.

Track and Quantify Process Improvement Impacts

Even though the initial costs of setting up CI + CD can be quite high, CI + CD saves more time for developers in the long run. We can track the amount of time saved with CI + CD over a project’s lifetime. A 10% improvement in build time, results in +99% time savings over 100 builds. Tracking time savings is easy with CI, and leverages the power of compounding. No wonder Joel says buy the best tools money can buy 😅.

Correlate CI Metrics with wider Team Metrics

We can even correlate CI metrics with other metrics like Agile team velocity. PMs can visualise key datasets to make faster, and better project decisions. A faster decision-making process helps to manage project complexity. For example, we can know when would be the best time to start spikes, respond quickly to changes of App Store guidelines and schedule backlog tasks by priority.

Track Technical Debt through Code Coverage

We can track changes to code coverage with third-party integrations. Code coverage reports can be reviewed to highlight files that need to be tested. CI helps us to refactor because we need to ensure the codebase is covered by tests before refactoring technical debt. As the project develops, we can also compare this to crash rate metrics to quantify the effect of better testing processes.

Can CI + CD ever be a problem? ❌

CI and CD are powerful tools, however, don’t use a hammer to crack an egg. Consider carefully how best to implement CI and CD on your project, and consider it as a long-term investment.

Long-Term Projects Only

CI + CD is only worth the effort setting up if the project is going to be distributed and maintained long-term. The effort to set CI up won’t be paid back over time for prototype apps. CI and CD are tempting tools to use because of their convenience, but we need to use the right tool for the job.

Essential Processes Only

Automate only the essential processes like testing, static analysis, and deployment. Automating everything under the sun is a recipe for hopelessly long build times. If builds are taking too long, CI is no longer convenient or helpful. You may need to add codebase security scanning for the client but streamline your workflow as much as possible. Furthermore, a configuration with CI but no CD makes more sense for some projects.

Do Not Over-Optimise

While it’s super gratifying to view a dashboard of project metrics, it’s important to only track the key project metrics. Don’t try to over-optimise everything under the sun. Balancing resource allocation with the frequency of CI jobs offers only marginal cost-saving compared to the amount of effort. There’s a sweet spot between configuration effort and business value that we need to get right on a per-project balance. Otherwise, CI no longer provides value for the business cost.

I believe deciding what metrics to track should be a group decision based on business and developer feedback. SonarQube code quality metrics are another example that may be important to developers but do not make sense to product. In these cases, it’s important for the team to communicate openly and reach an agreement. Ps. managing technical debt makes business sense too.

Track Effort Required for Maintenance

CI requires continuous maintenance to be effective. The cost of maintenance can be tracked over a project (e.g. the number of JIRA bugs tagged to CI configuration). CI makes metrics very clear, so we can quickly identify when tooling is not worth the maintenance effort. Maintenance effort should be much less than the setup effort, usually to respond to new announcements or APIs for less than an hour a week.

Choose The Right Hosting

Self-hosted CI requires more maintenance than hosted CI, however is more cost-efficient. On the other hand, hosted CI solutions are convenient but more expensive. Companies can choose the solution that works best for their project because there’s a trade-off between these two solutions.

In some cases, businesses won’t approve of either CI option. Hosted CI may not make financial sense for some teams, and we can’t set up Self-Hosted CI with lots of urgent deadlines.

It’s up to the development team to advocate CI makes good business sense, but management still may not agree. CI can be prioritised when the team has the capacity, but business needs often override development needs. As a developer, you can advocate for CI but often the initiative needs the buy-in from senior members.

Final Thoughts 📚

I hope you learnt a lot about the DevOps ecosystem for mobile apps, the benefits of CI and CD workflows and how best to implement it on your projects. CI + CD improves convenience with automation, adapts projects to change with a flexible configuration and improves transparency with metrics.

I am a big fan of automation, in fact, one of my mantras is:

“If it can be automated, automate it.” — Me

The best developer teams I have worked on have all had CI + CD configurations scaled for the production project size. I have never implemented CI and CD for a prototype or smaller project, unless for learning. CI + CD makes sense for production apps, especially if there’s a large backlog or development pipeline. Just bear in mind that implementing CI and CD is a long-term investment, and takes time to realise the benefits listed here.

Fig 3: How CI and CD worked for me in practice

NOTE: xkcd 1319 exaggerates the pains of setup in my opinion because automation has only taken a maximum of 2 weeks in my experience. Keep an eye on setup time though! Implementing a particular CI and CD tool may no longer make sense if the setup takes too long. Self-hosted setups are generally more work to set up, however, newer hosted CI tools like Xcode Cloud may not be stable in beta or early versions. In my experience, it’s unlikely to take longer than 2 weeks to get a basic workflow working.

Thanks for reading! If you enjoyed this article and want to hear more from me, please follow me on Twitter. Happy coding! 👨‍💻

Special thanks to Felipe Ricieri, Roger Tan and Sam Dods for providing feedback.