Achieving Continuous (Delivery + Quality)
At events and conferences, I am often asked questions about how we have been able to deliver value to our users quickly while also meeting or exceeding product quality standards. It isn’t easy. Our primary product is the digital platform JSTOR, a content rich site serving upwards of 12 million users in peak months. Many organizations struggle with defining and implementing a holistic quality strategy that supports and enables continuous delivery to support their products. Quality here at ITHAKA has not been magically attained; it is something we value deeply and each of our teams have taken direct ownership of making it a reality.
As I reflect on the state of our quality practices from 3–4 years ago, I recall long and painful manual regression test cycles for our quarterly and eventually monthly JSTOR releases. Throughout our sprints, engineers would complete story work and then a hand off was completed with QA. Each story was carefully validated after the coding was done. Prior to a release, multiple members of the QA team would spend anywhere from 2–4 weeks completing a manual regression test cycle. Only 15% of our platform tests were automated at that time. The number of defects making it out to our production website decreased; yet, we weren’t able to reduce bugs as much as we would have liked. After big releases were launched, it was not uncommon to have maintenance or bug fix releases. Tending to the issues post launch was no easy task. We cared about quality but we went about achieving it in incredibly inefficient ways.
It was time for a new strategy — but what would success look like? For us the following outcomes were very important:
- Faster response to business needs
- Business risk mitigation
- Product deficiency prevention
- Efficient and sustainable products & technology
- Scalable processes and practices
- Lower total cost of platform quality ownership
- The delivery of high quality applications and services
We rallied together to innovate and continuously improve our practices and culture while embracing team ownership of quality, the development and implementation of a state of the art Continuous Integration (CI) pipeline, and a full transition from manual testing to a more automated approach. The long days of manually validating feature/function enhancements are long gone, and it has been quite a journey.
Today we leverage both Test Driven Development (TDD) and Acceptance Test Driven Development (ATDD) practices. Often times, we have our automated tests ready before development is done working on a story so we move things through the testing process very quickly. Manual test suites are a thing of the past. Test automation happens at the story level and when we are making changes to the site, we are able to run the appropriate tests suites within minutes which has reduced our feedback loop to engineering significantly.
Just yesterday one of our teams deployed 23 applications in our test environment, bugs were identified and fixed quickly allowing the team to move their changes to production within a couple of hours. One important thing to note is that our entire QA team was a group of manual testers, all of whom now automate tests using Cucumber and Ruby. We have a team of coders! The transition was challenging, but extremely rewarding for everyone.
Change can be intimidating, but our teams engaged in this evolution and we are reaping the benefits today. Our learning never ceases — what was done today may change tomorrow, but for the better. We live in a world where we strive to continuously delight our users while also leveraging new technologies and enhancing the way we build with each new day. I’m looking forward to sharing more details about how we have achieved continuous delivery + quality!
Learn more about our team and becoming a QA Engineer at ITHAKA.
