Canary testing: What is it and why is it used?
Canary testing allows developers to test new software on a group of users before launching, helping to find and fix issues before they are deployed at a larger scale.
It’s no secret that new software feature releases often come with new issues. With software being as it is, and with developers working under the pressure of budgets and deadlines, it’s inevitable that bugs will find their way into the picture. Luckily, there are ways that software developers can help reduce risk and validate new software before public release.
Canary testing is one technique that allows production bugs to only affect a small group of users. By only rolling out a software release to a certain groups of users, issues can be found and fixed before a bigger release.
Why “canary”?
The term “canary testing” is based off of a mining tradition that was used up until the 1980s. During this time, canary birds were used by coal miners to detect carbon monoxide and other odorless toxic gases in coal mines. If the canary died, miners knew that they had to take action and evacuate the mines before becoming a victim themselves.
Likewise, albeit much less dangerous, canary testing in software can provide developers with early warning indicators if anything is awry.
So how does it work?
Usually, when developers are done coding a feature, they will test it for any general bugs. Once it passes these tests, the feature will be tested for issues in an environment with software testers or colleagues. Any resulting issues can be fixed and then implemented for end-users.
However, differences in production environments, such as operating systems, and storage and bandwidth capacity, can result in certain issues still making it through to production. This is where canary testing comes in handy.
With canary testing, once a release is ready, it can be deployed to a small percentage (e.g., 5%) of users who will be able to see and use the new features. By monitoring this user population, developers are able to identify and fix any bugs that occur, ultimately minimizing the impact of having production bugs in the software later on. This also means less users encountering the bug, and consequently less pressure on developers.
Canary testing at Synology C2
Synology C2 follows a similar approach to testing our software with canary testing. The first thing that we do is categorize users into three different groups before a software launch:
- Synology colleagues
- Early adopters
- Remaining users
These test groups are canary-tested in stages, starting with Synology colleagues, and followed by early adopters and all other users. Based on our algorithm and their respective data centers, the remaining users are randomly allocated into groups of varying proportions for each canary release. Features are released in small increments to users (i.e., from 1% to 100% of users).
By slowly increasing the number of users that have access to new releases, Synology C2 is able to monitor and capture metrics on the impact of the new version on different environments. If any issues occur, the infrastructure can be reverted back to its original state (a.k.a., the previous version) until the problem is fixed.
Many organizations also employ similar canary release strategies, but with different orientations and scales.
Canary testing in tech
Facebook uses a quasi-continuous release cycle to achieve rapid release at a massive scale. With this method, committed changes are batched together and pushed every few hours, making rollbacks easier and delivering code at a rapid rate. This technique is deployed following adjustments after internal testing. Canary deployment takes place throughout the production phase, and it steadily increases from 2% to 100% of production using a continuous monitoring system to detect issues.
Google, on the other hand, uses an automated release system called Rapid. Rapid uses blueprint files that contain build and test targets, rules for deployment, and admin information for configuration. Rapid also utilizes canary testing for each release cycle. Following the completion of systematic tests (e.g., sandbox environment), a small percent of tasks implement canary deployment, which are then followed by additional canary tests on large groups.
Through the use of canary testing and continuous release deployment, companies like Synology C2, Facebook, and Google are able to provide quality products and services.
Get access to the newest technology
Want to help test out new releases for Synology C2?
You can get access to C2’s newest features and give direct feedback to our product managers by signing up to be an early-adopter.
C2’s dedication to well-tested products
Synology C2 strives to keep our users on top of everything. By producing well-tested and high-performance solutions, C2 not only helps strengthen security but also makes sure our users get the experience they deserve.
More stories on the way…
Sign up for the C2 newsletter to get the latest updates on C2 services, products, technical insights, activities, and events.
Written in collaboration with Arden Lim.
