Accelerating Your Organization
Why continuous integration should matter to more than just your engineering team.
Continuous Integration is often seen as a Developer tool in many companies. The problem with that mindset is that downgrading Continuous Integration as only a developer tool actually ends up destroying its innate usefulness to your organization.
For those that may or may not have heard the term Continuous Integration, here’s a brief breakdown to catch you up.
Continuous Integration (CI) is the art of ensuring all changes made by developers can be merged into the master code base cleanly, by automatically testing new code and creating builds.
It’s a simple concept, but how this gets implemented can vary widely from organization to organization.
Here at Fuzz we have been using CI for the past two years. Over that span of time our approach has evolved, and we’ve learned a ton. There are three main takeaways we want to share that can make CI an efficient and effective organization-wide tool.
Nightly vs On-Demand
Our original CI implementation involved the system automatically creating new builds every night. Going from manual builds, created every few days, weeks or months, to this automated process was obviously a massive improvement to the whole team’s ability to see development progress.
The biggest setback was that the code wasn’t necessarily in a good place each night. Sometimes a developer would be in the middle of a particularly difficult feature. These WIP features would fail the CI test, and so our nightly builds were resulting in nearly a 50% fail rate. It’s tough on team morale to pass a test less than half the time.
From there, we adjusted our implementation from scheduled builds every night to builds that would be triggered on code commit. That is, when a developer was finished with a complete set of code, they would commit that code to the repository and our CI would be triggered to test and build. Then we extended the trigger to include merge requests, or when a completed feature was merged into the main code base.
This transition to on-demand CI testing and building saw a massive drop in failure rates, because now developers were ensuring their commits contained complete, working code. And we were still able to maintain our commitment to higher frequency and great accessibility into the development progress for the whole team.
Because there can often be multiple commits in a day, we actually saw an increase in the number of builds we were creating across the agency. Most of our apps get built about five times per day — or 100 times per month.
Side note: Ensuring a single test-and-build session can be executed in under 20 minutes is key to maintaining a commit-triggered approach. If that isn’t feasible, one solution might be to split your testing into a fast test suite (smoke test) and a nightly build that runs the whole project.
Team Rivalry
This brings us to the second takeaway: leveraging CI insights to improve team performance.
It was clear from the start that not all developers were being diligent about keeping their project in perfect working order throughout the development cycle. Remember, we had migrated from a mindset of cleaning up code before a weekly or even monthly build would go out, to maintaining clean code always.
To give the developers insight into the stability of the project, we installed a Build Monitor that was visible in one corner of the dev area. The screen would display the current status of each project, including whether the latest builds have passed cleanly, passed with some errors, or failed outright. Unless someone went out of their way, they would likely not even notice this engineering scoreboard. Thus, the impact was negligible.
The next evolution of this tool was to put up multiple Build Monitors, one for each engineering department at Fuzz. Each monitor would highlight the status of the department’s respective projects (iOS, Android or Web). Now that all engineers could see the health of each project, developers started taking extra care to keep their project in the green (signifying a “pass”).
This was the moment when we saw dramatic changes in how the teams interacted with the CI environment. Teams that had only a few failures suddenly realized they were so close to becoming a fully automated team, and made strides to button up their code. Teams that realized they were responsible for the most failures immediately starting taking it more seriously. Teams that were already working well, but whose test pass rate wasn’t 100%, starting working on improving.
It was at this point that we decided to take this one step further and move the Build Monitors into a more centralized location in the office, where everyone could see them. Now designers, product managers and executives could get an instant status update on the best-performing projects. Since then, we have been at 2–3% failure rates across all builds company-wide.
Delivery
The final takeaway is on delivery. Constantly building and testing your code with CI is great; providing insights into the testing is even better; but then making the successful builds available to the whole team is a complete system.
All successful builds are pushed to a distribution cloud, where anyone at Fuzz can access the latest build for all projects. That real-time access to projects for designers, product managers and executives allows for the entire organization to be tuned in to what’s going on at any given moment.
Under the Hood
Here at Fuzz, our CI Environment runs on Jenkins with a distributed system of workers for various project types that range from iOS, Android, and Web, using physical hardware as well as virtual hardware through Amazon’s AWS.
In the future, we’ll provide technical deep dives on some of these components and how they work. Hopefully these key findings that work best for us will also help your organization embrace and flourish using Continuous Integration practices — or choose Fuzz a partner in your endeavors.
The best products are those done quickly, correctly and with the full team involved.
This article is also posted on the Fuzz Blog.
All illustrations by Jon Shears, Product Designer at Fuzz.
