Testing Strikes Back

A long time ago in an environment far, far away…

At ClearScore our test environments have changed a lot over time. Over the summer, our SRE team developed a system that allows developers and QA’s alike to own their personal development environments; the environments are ephemeral, lasting for only the amount of time the developer needs them for, can have test-builds patched onto them and have our test-suite run against them. Before, we were sharing one or two environments between 90 or so developers which was getting very messy and complicated to manage.

Below is an outline of the journey we’ve taken so far. It’s far from complete, but hopefully our experience can help others avoid some of the pitfalls we found as companies scale up.

Rebels, Droids, X-Wings and Tie-Fighters: a history of test environments at ClearScore

2016 Team based environments (with STAR WARS NAMES!)
​​I joined ClearScore roughly 3 years ago and, at the time, each development team had their own testing environment. I was in the team named “Rebel Alliance” with the affectionately named environment “Y-Wing”. The other development team was “The Empire” and they had an environment named “Tie-Fighter”.

​​Each testing environment was configured to be as close to our production infrastructure as possible and allowed each team to test their changes in isolation. Whilst this meant that teams could test their own code without interrupting the flow of other developers, the actual infrastructure running each environment would slowly diverge from production and the upkeep became a bottleneck. The environments were removed shortly after as we moved away from Star Wars team names and into different, bigger teams with more developers. I miss the Star Wars names but not the waiting for one person to be done testing their thing before I could start testing mine.

2017 Environment in-a-box
The brilliant Stack-In-A-Box (SIAB) was born, built in-house by one of our wonderful developers. It allowed virtual docker containers to run locally, one for each of the microservices which were growing each day. This worked really well for a while. It meant developers had their very own environments to play with and branches could be tested without disrupting pipelines and releases. It was great! However there were some issues with it; first, having multiple micro-services running locally made your machine sound like it was about to take off. Secondly, trying to start SIAB would take a long long time, and thirdly, it was very hard to keep up to date - we were scaling very fast at this point and maintaining each microservice to work locally was taking too long. Eventually it became so out of date that is was unusable.

2018 Slack channel coordination
By this point we’d started using CI pipeline’s for releases, dev merges to master, a pipeline is kicked off, master is deployed to a CI environment followed by staging and then onto production with tests run in between. But we were also using CI as our main test environment; between 50 developers at the start of the year to 90 at the end. It was decided that we could use a slack channel to coordinate putting test builds onto CI.

There were a lot of problems with this; having to make sure nobody else was using the same microservice before sticking your test build onto CI, if another branch was merged into master while your test branch from the same microservice was on CI then it would be overwritten. CI became unstable, so were tests failing because there was a bug in my code, or because of another test build that was on there?

2019 Ephemeral environments
Work finally started on a new solution: ephemeral environments. What we needed was a place where each developer could spin up their own environment, running in the cloud (so their computer didn’t sound like it would take off), where they could stick their test build before merging to master and run a test-suite against it.

So what exactly are ephemeral environments and how do they work?:

• There is a Kubernetes cluster which is dedicated to ephemeral environments.
• When a user creates a new ephemeral environment, a new namespace is provisioned for the user in this cluster — this will take the format adjective-scientist (eg. friendly-edison). All microservices for the user’s environment are deployed into this namespace, along with other ancillary artefacts. When the user has finished with their environment, the namespace is destroyed along with everything within it.
• When the ephemeral environment is created, it will be created using the same versions of applications as are present in the target market.
• No data is persisted — these environments are ephemeral — this means that they are only intended for temporary use. You get a blank one every time, and you can’t keep the data.
• Ephemeral environments are automatically deleted when they are 24h old.

Simply, now all I have to do to start testing a microservice is stick one command into my terminal. I don’t have to check that anybody else is using an environment or waste time asking on a Slack channel, my machine doesn’t sound like its going to explode and although the Star Wars names are gone, sometimes it spits out a funny name like ‘tender-panini’ or ‘loving-sammet’.