When Should we Test on the Production?
Each time when I hear “testing in production” I recollect in my memory an episode from HBO series about Chernobyl.
However in some cases it is really necessary.
- Not possible to prepare a test environment (case from practice)
Some time ago I was a QA on data warehouse project. I was responsible for testing data transformations. Once I had a task to make sure that ETL job makes correct transformations and calculations for the various use cases.
The process called ETL (Exact Transact Load) aims to change structure of tables from formats like are used by apps (f.e. 3rd normal form with reduced duplication of data) to (dimensional model which is effective for fast generation of reports).
Unfortunately I could not generate fake dataset with all possible use case and we did not had real data on test environment for source tables. The solution was to test a job on the production. I isolated my testing on a separate schema and selected time when there was no huge from performance view jobs. So my testing did not break production data and performance of ongoing jobs.
Two my colleagues from other team had to test on production all the time. They had a real-time data warehouse loading methodology.
2. Complex to predict how the system would work before the production (case from Internet)
Recently I got to know about Loon innovative project. It has a goal to bring internet connection to more isolated places on our planet or on areas after natural disasters.
The technology uses a balloon flying 20 km up in the stratosphere and sharing the internet connection.
The ballon is filled with different volume of helium, which is lighter than air. This mechanism is used to control how the ballon can rise and go down.
The Loon predicts the direction of wind on different hight, it allows to keep the movement to proper direction.
At the moment there are flying many loons around the globe. We can even check how it works in the real time.
Loon is a real miracle on the boundary of aviation, IT technology and telecommunication. Such kind of systems for sure need testing on production because it is impossible to predict all conditions.
The loon team really did a lot of production testing in 2011. Read their journey of finding a proper shape, materials and investigating laws of physics in the stratosphere.
Testing on production is needed when it is not possible to prepare a test environment or to predict how the system would work.
However those tests should not affect our users. In modern days the testing on production does not mean delivering not stable code and letting end users find bugs instead of us. Your production testing can be isolated from end users.
Feature toggles, A/B testing, monitoring logs ( and quick rollback/ rollforward in case if something is wrong) are the tech mechanisms, which can help on this path.
Other Useful References
- Agile Testing Condensed by Janet Gregory & Lisa Crispin (Chapter 8: Testing in DevOps, section Testing on Production).
- Testing in Production, the safe way (and series of previous articles) by Cindy Sridharan.
- Best Approaches to test in Production by Ari Noman.
