Experiments in making my role redundant: from Team Test Lead to Team Member
There are few times in your career that you truly get a chance to reflect on the job you’ve done. Even fewer I would imagine that you leave a company with your head held high, proudly explaining that you achieved your experiment in making your role redundant. Classic questions such as “who will replace you?” and “how’s the handover?” that one would normally expect are instead replaced with requests to communicate how others can continue the approach at scale.
Therefore before my sieve-like memory forgets my pre-Unruly life, this blog recounts adventures in quality with mobile apps teams at my previous company and the journey on how I went from Test Team Lead to Team Member. Ultimately how I left my team to autonomously own their quality practices and to socialise their work, leading the way forward for the wider company and other teams.
Do we need dedicated testers for teams to deliver quality at speed?
The Experiment
If we remove [QA]/[Tester]/[Test Engineer]/[Test Lead]/[DevOp] as a siloed role and responsibility then the team will share the burden and together work to improve the quality of the app.
Let me set the scene... 7 teams, roughly 2:3 tester to developer ratio, 6 different apps across 2 platforms (iOS/Android) for 4 brands. The then Test Team lead had their role split to handle the workload — enter me. Surprisingly knowing very little about mobile app development turned into my biggest advantage, as I didn’t have any bias influencing my approach. First step was to help identify the teams challenges in quality, and what might contribute to their lack of confidence. Best thing about starting a new job: asking tonnes of questions. Always results in the most informative and honest answers.
The Challenge
After scribbling my way through 5 notebooks (I’ve since invested in a higher page per book ratio), the teams appeared to have a few key difficulties in the name of quality:
- Blame culture. App clients are bound to services, and it’s easy to blame another team for their service than it is to work together to understand the impact. With a fundamental breakdown in communication between teams, lack of architecture education, and no service contracts meant the teams pointed the finger at others.
- Role silos. It seemed easier to blame someone else for not doing their job, than it is to work together to ensure the job is done right. Having distinct roles in the team meant more finger pointing.
- Legacy code. Inheriting outsourced apps meant technical debt, severe lack of unit tests, lack of understanding, and ultimately a fear of owning the code. Developers had little confidence in making changes, and therefore had an over reliance on testers and E2E tests to catch issues. Easier to blame legacy code than it is to take ownership.
What did this all result in? Our customers having a bad quality experience.
The Vision
“If you are really good at it, they won’t even know it’s you.” — Joakim Sundem
The first steps to helping solve these issues was to curate a shared approach in which we could remind ourselves what we value in quality. Helping to support why should the team invest in quality. We collaborated and defined a Quality Manifesto for the team.
We are uncovering better ways of testing apps and delivering value to our customers, by doing it and helping others do it. Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.
The Process
The curation of the manifesto enabled us to have lots of constructive conversations as teams. It was a little bit of a game changer. Everyone started thinking outside the box. It never was, and never intended to be, rules to develop by. You could argue that that this manifesto actually could be interpreted as: we as a team will work together to ensure we develop what we understand to be the best implementation at the time; automate this understanding.
Through facilitated sessions with the teams, we drilled down into the definitions of each of these values. The intention that we iterate on these definitions as we progress. Understanding impact really summarises the ideal state whereby the team don’t make assumptions on what they are developing on behalf of users, stakeholders, customers or anyone with a stake in that feature. Understanding what risks, benefits, how to measure, and ensuring that the right thing is actually being implemented helps to drive a quality outcome. It also promotes communication outside of the team, a step closer to addressing blame culture habits.
The second and third values on not leaving all the testing to just before shipping, and automating repetitive tasks (perhaps naively) self explanatory and should be well understood across software development in 2018. But it’s good to reinforce all the same. Also questions what automation means. Is it just automating manual tests? How can we automate more infrastructure, deployment processes, Slack bots for user feedback, wrapping legacy code in tests for understanding…?
Reflecting on the last two values invokes the memories of when I was applying for Unruly. I shared an old Quality Vision presentation I had done with an ex-colleague of mine. He asked a very simple question:
“I see you mentioned T-Shaped. Have you considered working towards E-Shaped?”
While I don’t believe individuals should be striving to be defined a letter of the alphabet (I-shaped, T-shaped…), the term ‘E-shaped’ has been referenced since 2010, and the fundamentals of this concept really do help cement forth and fifth values on team responsibility and collaboration. The concept involves teams removing dependencies that could block them, and instead building a team of next-generation specialists. In theory, this can result in teams with limitless potential.
The Methods
One of most surprising contributors towards the teams embodying E-shaped traits was the changes in team structure. The absence of a person in a role would once be considered a breaking change, instead it became a gap that could be learnt and filled by the teams.
Over the course of a year we lost Android developers, iOS developers, PMs, BAs, Scrum Masters, designers, and UX. Not to mention other personal changes throughout the business. We mourned primarily for the loss of the people and their contributions, not for the role they were hired in. Within a year of starting, the ratio of testers to developers reduced from 2:3 to 1:4.
So how did we fill the gaps? We experimented with many different methodologies, to varying degrees off success. One such experiment was rotating titles (Batman — saving the team from technical distractions such as CI, infrastructure, flakey tests, server side changes. Demo-gorgan — demoing value to internal stakeholders every sprint. Release Captain — taking care of the manual side of app releases) ensuring that everyone upskilled in difference aspects of the day to day.
The teams started bringing more under their control, and naturally started progressing towards becoming E-shaped. They learnt how to use data to drive decisions. Financial impact, user impact and risk analysis was all learnt. They joined architectural discussions to ensure all clients were understood. They gathered business requirements, engaged in user research and UX sessions, wrote their own copy, and discovered new means of gather translations. They even started designing their own assets.
The Methods in Testing
As part of this experimental journey, I helped to work with the team to redefine the traditional testing pyramid. For more in depth technical implementation, Hesham Salaman wrote a fantastic Testing Manifesto that complements the pyramid extremely well for iOS development.
“Team of Testers” base layer — Exploratory Testing
- Test pre, during, and post writing code. From API creation, UI implementation, to A/B experimentation.
- Always test on a physical device. Customers don’t use emulators.
- Gather feedback from colleagues, stakeholders, customers, your family, your dog…
- … and understand their needs.
- If you are repeating it, automate it.
Network independent foundation layer — Unit/Snapshot/UI Tests
- Offline, offline, offline.
- TDD.
- All tests run against mocks, stubs, fakes, spies, or dummies.
- Regularly update offline stubs against dependent services.
- Test the stubbed data, don’t assume.
- Micro-features can be independently tested without needing to build the entire app.
- Feature integration should be covered by UI Tests that are network independent.
Integration layer — Contract Tests
- Consumer driven contracts work to document in code how the clients are consuming services.
- Tests run against all supported environments
- Using a collaborative, in-house, open framework to ensure full transparency
“The Icing on the Cake” layer — Red Route Tests
- Name stems from city planning road markings, see Red Routes.
- Identify the minimum number of positive journeys through the app that gives the team confidence in releasing. These form the definition of a Red Route.
- These could be driven by User Story Mapping.
- Run against Staging and Production.
- Ignore feature flags until they are released into production.
- Remove tight coupling of UI driven acceptance tests.
Taking the emotion out layer — Data Driven Testing
- The apps leveraged tools such as Firebase, AppDynamics, and Appfigures to promote fast customer feedback.
- Phased releases.
- A/B testing bug fixes. Utilising feature flags.
- Always run and monitored in production.
- This can be assisted by introducing Error Budgets.
The Results
There are 3 key results: quality, speed and number of dedicated testers.
Ask most people who work in software and they too will express how hard it is to measure quality. Especially when technology and tools change constantly. I determined crash free users as a solid metric that could measure customer impact. Both apps set an objective of 99% crash free users, they both met/surpassed it.
Cycle time decreased for both teams. What does this mean? Teams are delivering against their definition of done faster. What this graph doesn’t show, is that post Christmas these times reduced further.
Finally, the number of dedicated testers. As of January 2018 there were still those whom identified as testers or QA. Myself included. Shortly thereafter, the ratio of testers to developers shifted to 0:1, with team members identifying themselves as engineers. One tester measured his progress by taking the interview for a developer role. He got the job.
The Conclusion
Due to successfully making my role redundant, it allowed me to pursue new adventures with Unruly, working to contribute to the success of the developments teams and the company as an Agile Test Coach. Thankfully this new chapter begins with teams owning their own quality! Now I get to indulge in working with teams to continuously improve their current practices.
This experiment has cemented my belief that a great teams will always be better than rockstars in defined roles.
