Sleep safe at night…Test All The Things!
What is a good night sleep worth? Any new parent can attest that all they want is a good night sleep. The constant demands of a newborn can stress even the most well prepared parents. Compound months of interrupted sleep, and you have a person who is irritable and unfocused. Well, even seasoned investors enjoy an undisturbed sleep.
I will not trade even a night’s sleep for the chance of extra profits. — Warren Buffet
So, how does this relate to Android or iOS platforms? As the community matures, so do our responsibilities. With billions of devices and users consuming our applications, we are evolving into mobile 2.0. We are in a mature industry that demands more accountability, and wants continuous feature roll out.
In one of my past lives I worked in such a business. Millions of subscribers paying millions of dollars, yet zero automated tests. The team didn’t sleep well at night. When I asked about why we have zero test coverage, leadership told me “We have PagerDuty”. I watched as people were awoken in the middle of night, called regardless of religious holidays, bothered on vacations, and chewed out when they didn’t respond. Needless to say, the attrition rate was high. Since the company’s values didn’t align with my own, I too left.
Don’t just work hard. Work smart!
Inspiration
I pull quality testing tactics from one of America’s greatest industries, the military industry. We used techniques like writing software requirements, developing enduring architecture, deploying engineering quality in all things, and testing all the things.
Quality was as important, or even more important, than the amount of features we could develop. The common phrases surrounding quality were, “Planes don’t just fall out of the sky” or “When in doubt, test”. We all knew lives were on the line for the products we built. Quality was not something we did only in development, but also in post production to ensure full time readiness in a wartime theater.
The majority of the commercial world still consider this concept too expensive or time consuming. However, entering mobile 2.0 things have changed. Our consumers demand more. High efficiency companies are already testing in all phases. My goal here is to describe, at a high level, how some companies execute full time quality and therefore sleep safe at night.
Development Quality
The quality assurance diagram above covers a wide range of testing scenarios during the development process. As software development matures and companies embrace mobile 2.0, we are automating this entire process. We expect to find bugs within minutes or hours, and not days or weeks.
Unit Tests
There are many wonderful articles written about this topic, so I’ll be brief. Unit testing is a developers first line of defense for sleeping safe at night. Do this regardless of the culture. These are developer driven and are a daily part of our craft. Behavior test your code, don’t test implementation. Test public methods, not private methods. Mock when appropriate. Use robolectric sparingly, or drop it all together, in favor of a strong MVVM architecture that uses dependency injection. Start with strong tests around your business logic and work out from there. Be bold and create a culture of TDD.
Integration Tests
An integration test verifies the communication paths and interactions of a component to detect interface defects. From an Android perspective, a component can be an App or Library. The main idea is to connect modules/classes together and test those connections as a subsystem. Below we talk about black box and white box testing.
Black box testing takes an external perspective of the test object to derive test cases. The QA engineer selects or mocks valid and invalid input and determines the correct output. There is little to no knowledge of the internal structure. QA engineers seek to verify functional and non-functional product requirements. Typical tools used by QA engineers to automate this process are Selenium, Sikuli, Appium, and BrowserMob Proxy.
White box testing takes an internal perspective of the test object to derive test cases. It requires programming skills to identify all paths through the software. Typically, the engineer who develops the component will write the white box integration tests. Their goal is to verify the reliability placed on major design items. From an android perspective, here is where we apply espresso testing, android instrumentation testing, and contract testing.
Contract testing is a test at the boundary of an external service that verifies it meets the contract expected by a consuming app. In other words, we test an Apps consuming APIs. Trust and verify those servers!
End to End Testing (E2E)
End to end tests is a methodology to verify that an entire system is performing as designed from one end to another. The goal is to identify system dependencies, and ensure the right information is passed between them, so they function as expected. Automated tests are wholly owned by the quality department with a focus on functional testing. The same tools are used as before, but no mocked inputs in this phase.
Production Quality
With the introduction of CI/CD via trunk based development, we now have the capabilities in the Android and iOS world to write quality assurance tests in production.
Feature Deployment Tests
Once an app is in the wild we can still perform tests to see how new features are utilized by our user base. One sort of test is a canary release test. This technique reduces feature risk. It does this by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure. You can accomplish this type of testing by using a concept known as release toggles (AKA Feature Flags).
Another type of feature-deployment test is champagne brunch test. This technique turns on new features for a set of internal or highly critical external users. This is an opportunity for them to drink the champagne early. For external users, imagine targeting a specific group of people, like the 20% of your user base that provides 80% of your business. These are our most important users, our Alpha/Beta users, and we want them to join us in the celebration! We accomplish this type of testing by using a concept known as permission toggles.
Experiment Tests
Also known as A/B testing, experiment tests allows product owners to test aggregate behavior on a different cohort of users. It’s commonly used to make data driven optimizations over the app’s usage. This differs from feature deployment tests, because the focus here is refinement of an Apps usage instead of acceptance of new features. We accomplish this type of testing by using a concept known as experiment toggles.
Summary
A well disciplined engineering culture, with the proper leadership, will have quality engrained in all the things. Quality engineers will be involved in product requirements, planning, and retrospectives. Testing would be bundled in for every sprint planning where every story point estimation would also include time for writing tests.
I like to sleep safe at night knowing changes by any team member, at any time or any place, will not break our work flow. No more late night calls for this engineer! So, I urge you to work and develop these cultures at your own company. Even with weak leadership, as an individual craftsman you can slowly point your stories so you can write more tests…a professional evangelization. Keep the discipline high for quality in all things. Your users demand it!
Feel free to follow me on Twitter, and Medium.
~LEO
A special thank you to all American engineers, past and present, who keep U.S. sleeping safe at night.
