On skills and roles in software development — automation
A recent blog post from Alan Page prompted an interesting subsequent twitter conversation.
In Alan’s blog, he writes:
Automation works wonderfully for short confirmatory or validation tests. Unit, functional, acceptance, integration tests, and all other “short” tests lend themselves very well to automation. But I think it’s wasteful and inefficient to have testers write this automation — this should be written by the code owners. Testing your own code (IMO) improves design, prevents regression, and takes much, much less time than passing code off to another team to test.
That leaves the test team to write automation for end-to-end scenarios. There’s nothing wrong with that…except that writing end to end automated tests is hard. I think that most test teams spend half of their time writing automation, half of their time maintaining and debugging automation, and half of their time doing everything else.
This mirrors my own experiences very closely and so no doubt feeds into my confirmation bias quite neatly. If I had to sum up my take on the subject, it would be ‘most testers would be much better served helping career programmers to write automation than attempting it alone’. (Speaking of sums, I believe Alan’s use of halves here is deliberate 😛 )
I had an interesting twitter conversation with Angie Jones (and eventually some others) in which we shared some differences in our experiences with test automation and who should be doing it.
curious as to where you predict the developers will get all this extra time from? ime, test automation is a full time job
- Angie Jones (@techgirl1908) August 15, 2017
Talking about pros and cons of automation in abstract is both difficult and for those like me who are lexicographically deficient, an exercise in futility. So instead, here’s a story.
The Falcons
The first team I joined at eBay was a team called The Falcons. They were some of the brightest programmers I’ve ever worked with. I’m going to name names, because if you ever get the chance to work with these people, I highly recommend you grab it. Steve Bennet was our very capable team lead, we also had Tomek Lewicki , Radek Scheibinger , Francisco Ferreira and me. Up to this point, my SOP when joining a new development group had been to assess the degree to which I’d have to coach people on the value of software testing. With the Falcons, I was immediately impressed by the importance the whole team placed on good testing.
They used TDD as the standard way to write code — and sure that’s not really about testing, it’s about driving good design using a unit testing framework, but (after deleting a bunch of stuff that no longer serves you) you do end up with a set of tests you can audit to see what is useful in a regression testing/monitoring suite. They wrote GUI tests that exercised the code they wrote in a sensible, maintainable way. By this, I mean one, maybe two tests that ran end-to-end that we infrequently ran. The majority of the tests used mock data to render the visuals we were creating. The business flows we tested under the hood where that stuff belongs. In case it’s not clear, we also did ‘manual’ testing to make sure these things married up.
Test automation was written simultaneously with production code. They were considered inseparable, not just by the team, but by management as well. A story wasn’t done if it hadn’t been tested, and for the vast majority of stories, that meant unit testing, integration testing and maybe even GUI testing in addition to exploratory testing. This was factored into any estimate we were asked to give. If there was a crunch period, we dropped features, not testing.
I don’t want to pretend that the team was perfect and there were never any problems, but those problems had nothing to do with testing or software quality being out of scope for programmers. The entire team owned how automation was done and the developers wrote the lion’s share of automation code. I paired with programmers on both production and automation code. We had our share of test failures and non-deterministic heisenbugs to quarantine and hunt, but we did it. Admittedly the crew weren’t as keen to pair on exploratory testing work, but most of the time I didn’t need them to. The project that we built ended up being adopted by a large number of teams across eBay and to my knowledge is still widely in use.
There’s more to the team’s success than just the team though. The management group made it their job to create an environment where the teams could excel. Yes there were delivery pressures, there always are, but it was recognised up-front that software development is not a conveyor belt of widgets. It requires the space to experiment, fail, learn and adapt. Of course we try to get it right the first time, but we recognise that there will always be hidden complexities that make that difficult. For that reason, the way new products came to the teams for development was carefully managed. No death marches here. We had some crunch times where we needed to pull some overtime, but this was never allowed to become chronic. More often if we didn’t hit a sprint commitment or a deadline, we’d stand up in front of our stakeholders in the demo and talk about it. The Falcons were an awesome team in large part because they were given the space to be.
This was the model that we used for all the teams in eBay’s European Product Development department. Some teams did (and still do) this really well. Some not as much, but team ownership of automation can work and can work exceedingly well in a sustainable way. For those that are curious, Steve Bennett and I spoke about the Falcons at Øredev in 2014.
I want to echo Angie’s sentiment that test automation is difficult. I think that fact is lost on a lot of people who get into it. They either don’t yet have the skills to recognise how difficult it is, or they’ve been hoodwinked by some snake oil salesman telling them how whatever tool they’re peddling will solve their automation headaches. Doing test automation well requires exceptional skills in both test design and programming. That’s a far less common combination than you might think. It’s far more likely that you’ll get this combination from a tester with strong test design skills pairing with a talented programmer.
A case in point, my own coding skills aren’t terrible. Up to the point where I’d joined the Falcons, I’d done things like rewrite a great deal of QTP’s back end to act more like Michael Hunter’s .net automation framework. I’d written a ruby automation framework that was modelled in part on Wade Catron’s efforts at Linked In . I wasn’t a complete numpty in front of an IDE. Watching the programmers in the Falcons operate though, I saw a level of skill that was miles beyond where I was.
They’d discuss appropriate patterns to use, nitpick proper structure of RESTful API, how to make sure classes and their methods could be testable and maintainable and where to test different things. Ever tried writing tests for a monolithic method that had a bunch of different exits under different conditions? Do you know what a giant pain in the arse that is? That didn’t happen, because the team that was writing the production code also wrote the tests to exercise it. Likewise I didn’t have to mess about with business logic to test aspects of page rendering as the guys had set up a test harness around the front-end to render any page as needed.
What I want to see are software development professionals (of all specialities) who have a grounding in the fundamentals of their peers. This is where I’m coming from when I’m saying that programmers should be writing test automation. Not that software testers shouldn’t at all, but unless they’re also an exceptional programmer, they should be pairing up with career programmers to do it. If you’re a tester who is also an amazing programmer and you can fly solo, then congrats. You’re awesome. I wish there were millions more of you. I’m not holding my breath for that eventuality though.
One last thing I need to address — I did suggest to Angie in that twitter thread that perhaps her talents were being underutilised. In hindsight, I can see precisely why that looked like I was saying ‘You’re wasted as a tester, you should be a developer’ — Apologies Angie, that’s not what I meant. It’s possible you’ll still disagree with what I did mean, which is fine, but as someone who has been on the receiving end of multiple ‘oh, you’re wasted in testing’ comments, I get how insulting that can be.
The most productive I have felt was when I was pairing with a programmer to simultaneously write production code and test automation. There is a consideration for both roles that happens because production code and test design happen in parallel that I don’t find when these are done in isolation. In that respect, my comment to Angie was an inquiry about the opportunity to share excellent test design skills with fellow programmers as well as teaming up to write both production and test code simultaneously.
That will do for this post, I think. There’s more to say, but less about automation and more about education in software development in general.
Originally published at http://testjutsu.com on August 21, 2017.
