Should we include Automation as part of Definition of Done

Following up on a provocative talk by Angie Jones at testbash philly on How to Get Automation Included in Your Definition of Done, I want to collect my thoughts here.

Coming up a commonly agreed up on definition of done is quite hard and each stakeholder has different perspective. At the outset it seems quite obvious and achievable but software development is complex and there are many exceptions and context of each story matters.

There have been arguments to include Automation as part of this list and in fact I have advocated for its inclusion in my team couple of years back. I lost then but thinking back it is a good decision because at the time of that project we weren’t setup to have automation in every story.

Lets talk about the challenges to start with

First, I work in mobile apps Android/iOS apps have a very different ecosystem than web apps and much different test frameworks. They are not as matured as web and take more effort to create reliable tests.

Second, there are quite a few stories that don’t require UI validation. e.g: infrastructure work, bug fixes etc.

Third, having automation in DOD will skew us to have a large number of UI Tests breaking the test pyramid model, if possible we should push tests down the stack. If a verification can be made through unit/functional level we should prefer that.

Fourth, SDETs/Automation engineers/Quality engineers are already pretty bogged down with in a sprint to do testing and having this automation requirement will only bottleneck the team.

Fifth, what about breaking/flaky tests? In spite best of our intentions and efforts tests do fail, for non-genuine reasons like infrastructure, devices, bad tests, intermittent failures. It takes a non-trivial amount of time to diagnose those failures and fix them. Should we add a story for fixing each one of these?

Sixth, apps UI designs change constantly before initial release and during major releases, creating UI tests for these take non-trivial amount of time and re-work doesn’t justify creating tests for each iteration.

Seventh, test infrastructure maintenance: it is not trivial to setup and maintain all the CI/CD setup parts. e.g. For my current project I have 5 jenkins jobs, my tests run on upto 6 emulators and 7 devices in parallel, scripts to upload, pull down results, create monitors and so on. Testops is real and time consuming.

For some of these I have now found solutions

Test automation tools: For android use espresso and for iOS use XCUITests there are multiple benefits of chosing a tool set that is supported by their creators and the language of the devs. These are first class citizens in their respective worlds and are getting matured like selenium.

Move automation code to same repo as devs: This one simple move has drastically changed how much my current set of tests are accepted and used accross. Devs are able to contribute/maitain tests, I am able to reuse helper methods easily and it is much more easier to setup CI workflows.

Focus on creating a framework rather than tests: Another valuable lesson that I learned from my previous project was to focus on creating a framework first and then adding tests. Logging, tasks runners to invoke tests, running parallely, jenkins jobs all of these are needed anyways and devs will help easily if these are already in place. Also this works will take few sprints and gives a chance for the UI designs to settle down(6)

Make devs responsible for UI Tests as well: Being in the same language, same repo as dev code are the first steps. Then give brown bags after a few tests. Finally be proactive in PRs and pair with devs to add UI tests as part of them. Then slowly one would acceptance of these tests along with the unit/functional tests.

Challenges that I still do not have answers for

Time management: Even in an ideal world with devs writing most of the tests, none of the tests being flaky, tools, infrastructure matured enough, there is still a lot of stuff that needs to be done apart from automation and we haven’t scratched the surface about non-functional testing. How do we manage time between testing and automation?

Test infrastructure maintenance: May be someday I will be able to get it to stage of set it and forget but for now, I am still having actively work on infrastucture.

So, at the end of the day, the answer is ‘it depends’ but now you few things on which it could depend on and may be you have a solution for them. For this project I am still averse but for the next one, I may be ready.

Thoughts? Any other challenges that I am missing?