You did it, you got an initial test automation effort off the ground! Maybe you even followed some of the steps I outlined in my article “How to Start Test Automation at Your Company”
The thing that’s probably hitting you in the face right now though is the realization that your coworkers aren’t psychic — they don’t really understand what you’ve built, and they forget that it even exists most of the time. It hasn’t sunk in that they can use your suite to tell almost immediately after a deployment finishes that everything worked as intended, and everything is good to go.
That’s frustrating, and even worse, it stings.
You’ve built this wonderful tool from the ground up! Why don’t people love it like you do? You start to wonder how you can get people to rely on what you’ve built.
I’ve been in those shoes, and I can tell you the answer: CI/CD pipeline integration (if it’s done right).
Why you need to make a change
Up to this point you’ve probably been running your test automation ad-hoc on your local machine and telling people what the results are. While that’s fine to get started, it doesn’t drive long-term value because:
- Your machine is only available to you
- Your machine is unreliable for a number of reasons
- Your runs have to be kicked off manually
- Your run results aren’t automatically published anywhere
Notice how every point in my list started with “Your”? That’s the problem — this is your thing and you need to make it our thing, comrade. Integrating your test automation suite into a CI/CD pipeline is the way to solve for these issues. Your test runs will become highly visible, the results easily accessible, and by extension, your test automation will finally be valued.
Getting started
One assumption
Before we continue I’m going to make one assumption, which is that you/your company already has a CI/CD release pipeline stood up and working. If you don’t, you need to get with your engineer friends and implement that first before you can do ANYTHING deployment related in a scalable way.
Standing it up
Now, first things first — make sure you have your test automation code checked into source control in your own test automation repository.
From there, stand up a dedicated VM (or whatever device you need) that the release pipeline will target to run your test automation suite. Make sure it mimics your customers’ most common setup as close as possible to get the most value out of it. It is also imperative that you follow all good housekeeping rules for the new VM — it should be highly available, backed up frequently, and patched regularly.
Once that is stood up in a maintainable way you should pull your repo down and get the VM configured to run your tests. Make sure they run successfully against all environments before checking this one off the list.
After that, you need to add a stage in your deployment pipeline to kick off and report on your test automation runs. This stage should be after your app has built and unit tests have passed. I’m not going to get technical on the actual integration as there’s too many different frameworks, platforms, etc. to be genuinely helpful here, but most tools and automation frameworks have great resources to help set this up. If you get stuck your engineering friends are a great resource. Once you think you have it working, kick off the stage a few times across environments and make sure that test results end up back in your pipeline.
Once that’s all done, congrats! That wasn’t easy, and I’m sure you want to tell the world.
Don’t. For real.
Breaking it in
But why can’t you tell people? You’re trying to make it ours, comrade!
The reason is that it’s imperative to live with your newly integrated test automation suite for a while to get a feel for how it’s going to work long-term. All too common at this stage is an inordinate amount of flake. “But it works on my machine!” you cry. It doesn’t matter, what matters now is how it runs when the pipeline kicks it off, so figure out the root causes of flake across your suite, and fix them. Once you’ve built up confidence that flake is all but eliminated it’s time to tell people.
Spreading the word
The best way to do this is to go back to whoever manages your pipeline and add a task to publish your test results to whatever asynchronous chat platform you use — Teams, Slack, whatever. Just make sure they get posted there somewhere where people will consistently notice it. MAKE DAMN SURE THE TEST RUN IS SUCCESSFUL THE FIRST TIME THIS HAPPENS.
The game now is changing others’ behavior, and they’ll only do it if they can trust what you’ve built. First impressions matter and if your test run publishes a failure the first time anyone sees it, you’ve got a steeper hill to climb.
When it runs the first time (successfully), @ mention everyone and call attention to what you’ve built. Tell them this is new and exciting, and it’ll run every time a deployment happens. They can be confident the deployment succeeded almost immediately after it’s finished. Keep calling this out for awhile until you notice behavior shifting and people begin to rely on your test results as the “all clear”.
Now, eventually your test run will publish a failure. Someone should notice that something failed and ask what’s up. Be glad they did, and fix it immediately. Someone noticing a failed test result and saying something is crystal-clear signal that they find value in the tests, but you cannot let failures persist. If you do, people will lose trust in the suite, and once trust is lost, it is next to impossible to get back.
Wrap up
If you’re persistent over a long enough period of time, people will remember your tests. Hell, they’ll seek them out. You don’t get that stinging sensation anymore that people don’t value what you’ve built — you’ll instead get the warm fuzzies the first time you see or hear someone say “Well the tests passed so everything should be fine.”
At this point you’ve done good. You built something from the ground up that people care about and find useful. You now own your own mini product, and will need to be cognizant that you maintain your own mini market fit. Keep adding tests and call it out to the team when you do. And for the love of God, keep your test cases updated to reflect what you do and don’t have automated. Hopefully you’re not using Excel.
Thanks for reading! If you liked the article please share it or interact with it in some way.
