Building a Quality Quality Assurance Team
Joyce Hon
Winston Churchill once said “Success consists of going from failure to failure without loss of enthusiasm.” Here is a story of achieving great success through failure— the story of Yammer’s QA team.
The Past
Before my time, there were a few attempts at QA at Yammer. There was a guy who tried to go straight into writing automated tests, but failed to get traction. There was a team of outsourced testers who bashed on our app on a regular basis, but the lack of collaboration didn’t work well with our agile development. Then I got hired to give it a try, with only a year of QA experience under my belt. Ailin, a receptionist at the time, joined the team as well. She didn’t have experience QAing, but was always eager to help engineers with the weekly deploys. (Also, she’s hilarious and a go-getter.) Together, we helped get the Files and Notes features out the door, which was a huge project that took many months, a couple dozen engineers, a couple weeks of war-rooming, and way too much crappy takeout food.
We realized that we were pretty awesome, so we started hiring like crazy, and soon enough we had six QA people who were manually testing projects and supporting weekly deploys. But this wasn’t enough!
The Birth and Death of the “QA Tools Team”
We hired some people who would build our test framework to write API integration tests / UI automated tests. We called them the “QA Tools Team”, because their job was to build QA tools. This was great for a while because they got to do heads down coding work and had the technical background to do this, but over time, the lack of collaboration between the two teams wasn’t working out. People writing automation didn’t know enough about the needs of those testing projects manually, and the people testing manually weren’t growing technically, even though they had the desire and ability. So we merged the teams. And with our powers combined—like the multinational teens of Captain Planet—we were much more powerful and effective.
One QA Team
With the merging of the teams, the people previously on the QA Tools Team knew it was important to build the testing framework we had in such a way that anyone on the team would be able to write tests and diagnose failures. We went through a couple iterations trying to get things just right, deprecating two entire frameworks in the process. Today, we’re able to say that everyone on QA team (and some developers too) have written tests and used our dashboards to see if new code has broken our core functionality.
Our vision for the team has changed a lot over the last two years, so we came up with a cool mission statement to make sense out of what we’re doing:
“Enable engineers to ship features with efficiency and confidence while providing the best end-to-end user experience across all platforms. Accelerate engineering productivity through gray-box testing, tooling, and robust automation frameworks.”
Future Failures and Successes
Work is constantly being done to get us more test coverage and make our tests more robust. We’re also working on alerting, getting IE and mobile coverage, screenshot comparison, and being able to diagnose bugs faster. The teams’ insatiable appetite for learning has also produced something we call “QA internal projects”, which is a rotation of 4-6 weeks where a small set of QA engineers get to work on coding a tool that engineering can use.
The tool we’re building now is a Rails app for updating and tracking updates to our A/B tests. It’s been going very well, and even though the team is sort of new to Ruby/Rails, they’re learning a lot and will be able to spread their knowledge to their team and future projects. We don’t know what project the next rotation will work on yet. Hell, we don’t even know if this rotation thing is a good idea. But if history repeats itself, whatever comes next will most definitely be awesome.
Joyce Hon is Owner of Sugar at Yammer. Also, SDET Lead.
