Software Quality Assurance @ Stratio Automotive
or “Tooling for the Dev Team”
--
Author Background
I first started working as a Software Quality Assurance Engineer at Stratio Automotive in October, 2018. It’s been a long journey since then and in this article I’ll tell you all about it!
Before we begin, I can tell you that when I first started I had 2 years of experience in CI/CD configuration management and Test Automation. Nowadays I’m involved in pretty much everything that is related with Software QA, including processes, infrastructure, planning and testing.
That said, let me be your guide to the exciting world of SoftwareQA@Stratio.
Brief Introduction to Quality Assurance
I don’t presume to have the ability of explaining Quality Assurance in such few words that it wouldn’t make a collection of articles. However, I do believe that a simple enough explanation of my vision of Software QA can suffice in giving you an overview over my own bias on the matter and help you get something out of this article.
Developers will develop;
Testers will test;
QAs will … QA?
Software Quality Assurance engineers will automate the tests. They will also implement (preferably automatic) processes that provide feedback to developers about the status of those tests and the quality of the code. They also setup processes and tooling that provide visibility of these metrics to the whole team. Manage the configuration of testing environments and feed that to Software operations teams. And also create and maintain the processes and mechanisms for releasing and deploying tested Software.
This incomplete listing of responsibilities applies to the scope of Software. However, Quality Assurance is (or should be) present in all the stages of the Software development lifecycle, assuming many faces at each stage. To be more precise, another attempt at a list of responsibilities follows:
From this short list (emphasis on short), it is obvious that Quality Assurance is not something that is the responsibility of one team in a company. As a matter of fact, QA is the responsibility of everyone involved in the process of developing a product. In the case of Stratio, the product is a hybrid solution that mixes data analysis, software development and hardware development.
The question then arises, what is the role of a Software Quality Assurance Engineer? Well, that’s a question I face every day and to which I can give you my own perspective. That’s what the next section is about.
Quality Assurance in a DevOps World
In the past few years, a methodology has been on the rise. As foggy as it still may be, it’s called DevOps and the industry seems to like it. Quality Assurance Engineers who claim to follow this philosophy tend to also advocate the so called “Shift Left”.
Try pressing “Shift” and then “Left arrow” to see what happens. Nothing? =)
Consider the following software development life cycle:
Shifting left is to enable and perform as much validation as possible in the earliest stages of this lifecycle. The idea is that this way issues are caught earlier in the pipeline and can be handled more cost efficiently.
In the concrete case of Software Quality Assurance, I myself find that my responsibilities are well described as follows: making sure that everyone involved in the software development life cycle has everything they need to ensure the quality of their work as fast, fluidly and correctly as possible.
Below is the commonly seen picture that comes up when you search DevOps on any Search engine:
What the “shift left” advocates propose is that each step is also a loop in itself. A loop that is comprised of validating, analysing (analysing failures especially) and improving. Analysing failures is a rather important part since they can be great teachers. The difficulty of analysing failures is that they are usually only detected in later stages than the ones that introduced the errors. Thus, it is critical that feedback from all stages flows backwards to previous stages though root cause analysis to provide visibility to the all relevant members of the team. Process failures come in many flavours and they are always great opportunities for learning. From bugs to clients whose needs have not been satisfied, the company must be aware of its failures in order to fix them, learn from them, and move forward.
All in all, not that much to do in QA unless we start to get creative! And that’s what we’re doing at Stratio. Let me tell you all about it in the next section.
Software Quality Assurance @ Stratio
When I first came to Stratio the culture of Quality Assurance already existed. People validated their work as much as they could with the tools they had in hand. I’m especially focused on the Software Team, but this applies to the whole company. However, few explicit processes were in place to validate the solution, and thus, QA was coming short in many ways. Well, challenge accepted! I’ll print you a diagram for each step taken so far in the direction of our final plan for QA in Software. The first one below depicts the state of affairs as I initially found it.
My first question as a QA Engineer had to be “What kind of problems are happening in production and why are they happening?”. Fair question, I believe, when you’ve been hired to close the lid on the bugs that are leaking out from Dev to Production. Therefore, a simple form was introduced that Devs would fill out every time they fixed a bug. The form was based on the “5 whys” technique for finding root causes. And here are the results for the first month (December 2018).
* Note: “UAT” refers to the lack of an environment for testing and “Automated Testing” to the lack of automated tests.
Well, as you can see, there was nowhere to validate the code before sending it to production and many bugs were attributed to the lack of a User Acceptance Testing environment. Moreover, the product team was in need of improving their game. Also, something had to be done about the quality of the written code itself (i.e. static analysis) since there were bugs being caused by code duplication. And last but not least, some of the bugs could have been easily detected with automated testing.
Whether this form was a good assessment of the root causes, I don’t know. But I believe it was a good starting place for the Software team, using concrete examples of bugs (failures), to expose their pains and somehow prioritize their needs based on actual data.
At this point, the development process (from the QA perspective) changed a little bit.
The strategy for the moving forward was and still is to take advantage of the team’s “will to validate”. Who best to validate what has been done than the people who developed it and the people who designed it? Therefore the first step was creating User Acceptance Testing environments and deployment pipelines in Jenkins that would allow anyone to validate the software solution at any given version.
With the new environments and an easy way to deploy the software, the process changed again. Developers began deploying to UAT after considering that their work was ready for the user. Then the product owners were notified by the developers and validated the changes according to the requirements they elicited. Finally, the definition of done for user stories and tasks was changed to only become completed when the PO approved the work.
Not too bad for a start, but we can surely do better, and that’s what we did. Having Jenkins setup and ready to execute CI, that quick win could not be missed. Also, having the knowledge that code duplication, as long as other problems easily detected with static analysis existed, SonarQube could also be added to the stack with possible/probable return on investment. Hence, the process changed again, this time to include Continuous Integration and Continuous Inspection.
Awesome! But wait!? We have continuous integration and no tests to validate our Software? Well, we should definitely start writing tests! At least unit and integration ones. That’s what we thought, that’s what we did. But first, a brush up on how to write SOLID compliant and testable code. That is what followed, a series of four talks promoted proactively by the whole team in order to remind ourselves of how to write high quality and testable code.
At this point in time we also realized that we were using outdated, soon to be deprecated technologies in our code base. That, along with a sizable amount of technical debt, lead us to the decision of starting the migration of our codebase to .NET Core and Angular 2.
Well, that couldn’t have gone any better could it? It was hard work (and it is still ongoing), but indeed we made a good decision and all the necessary efforts to make it work. Writing this text from the future I can now say that. Let me just add that it was only possible due to an amazing team that really went above and beyond to learn and improve itself.
These changes had a lot of impact in the development process. However, I won’t redraw the diagram, since I would only be adding one new phrase: “Writing Automated Tests”.
At this point in time we started to have a lot of libraries, and some delivery processes had to be put in place. We opted for having manually triggered delivery pipelines in Jenkins that bundle the libraries and send them over to a Nexus Sonatype server. The artifacts are versioned with semantic versioning and optional labels associated with development releases.
At this point I consider that our QA processes and infrastructure were enough to meet the needs of the team. Therefore, it was time to start optimizing and improving what we already had. Enter the development of the Automatic Tests Framework. This framework is a tool that allows both developers and automatic system tests to launch Stratio’s services and core tools (i.e. the whole or a part of our Software platform) in any given Kubernetes cluster. This way, developers are capable of launching their code during the development stage and validating it in operation. Moreover, automated system tests are capable of easily launching, configuring and interacting with these same environments. Again, the same thought pattern applies: enabling everyone to validate their work as quickly and correctly as possible.
And that’s where we’re at right now. The ATF has proven its value a couple of times, and will hopefully continue to do so as more and more features are developed. Moreover, even though the consistent automation of system tests is still on the horizon, when the time comes this tool will be instrumental. Last but not least, our UAT environments can now be launched on demand with this tool, allowing us to move away from costly Cloud solutions. Our current process looks a bit like this.
Final Word
At Stratio everyone cares a great deal about quality, and that’s the only thing that makes these processes work. Try having Sonar and not having any developer looking at the issues it reports. Try having Continuous Integration and no one looking out for broken builds. Try having releases and developers duplicating code anyway because it’s “faster”. It simply won’t work.
And not just the developers are committed to the quality of their work, everyone in the company is just as much. That’s the only way major refactors of the code base are possible, that’s the only way that investment is made in quality.
I would like to leave a final thought, one that is mine original. Since I was first challenged to come up with an approach for Software QA at Stratio I took one entity as my client. That entity was the Software development team. My goal from the beginning has been to work towards improving the team’s efficiency and efficacy. I’m not alone in this effort. There is IT doing the same, there’s a Scrum master doing the same and there’s a product team also concerned with that. At the end of the day, it is the everyday work of a capable and united group that brings about truly amazing stuff. And what a journey it has been!
A final piece of wisdom: “it is testable, therefore it is”.
