How did Yoshi help Mario speed up our Deployment Pipeline
At Outbrain, we work in a complete micro-services cloud environment.
As part of the deployment pipeline of an Outbrain service/app, we have a pipeline manager we call Mario, that defines instructions on how to perform the service’s deployment pipeline, and executes the instructed phases, while updating about the progress on a dedicated Slack channel to give engineers visibility into the deploy & test process.
This has worked, and is still working great for us. But we always strive for improvements and efficiency, while providing a better developer experience, and we began to notice that there is a lot of room to enhance the process. We figured that a great step forward could be achieved by reducing the number of automated tests running as part of the pipeline, and only focus on the relevant ones modified by the developer who pushed the changes. Essentially, this means that we can cut some automated testing suites out of the pipeline, those that are irrelevant to the last committed and deployed code changes.
First Attempt
Our initial thought and first attempt at achieving this was by trying to filter out the web application’s components that were not modified by the developer, and in turn, not running their relevant automated UI/E2E tests. In order to do so, we would have to keep some kind of index that maps between a test and the relevant components it is navigating through when running, so that once a code (or a component/file) had changed, we can use the mapping to decide which tests are relevant, and only initiate those. This can be done using a code instrumentation technique, and can tremendously reduce the number of running tests on each deployment pipeline.
While experimenting with this method and starting to build an initial POC, we soon realized this solution is impractical for 3 main reasons:
- Extracting specific tests out of a few test suites means we might be too focused on what we’re testing — which is good — but we also might miss out on regressions that the commit had caused.
- This method takes too much work to maintain, is prone to errors, and over time, may offer less value than effort.
- Instrumentation is applied only on the simulated version of the app. This creates another element of inconsistency between the production and simulated versions.
Zooming Out
An alternative method, which was the one we actually chose, leverages an internal Outbrain implementation of Serverless. The idea was to extend the already-in-use Mario Pipeline Manager that was mentioned before, and give it an additional configuration file — yoshi.json, as we named it.
Now, our deployment pipeline has an extra step, right after merging the new PR, and before initiating the deploy & test phases. This creates the magic of dynamically modifying the initial Pipeline config, and reducing the number of automation suites running to only the relevant ones. It does this by fetching the modified app files from the last PR in Bitbucket, passing them through the pre-defined Yoshi rules the team created, and is located within the service, to exclude suites out of the pipeline, if the modified files are defined in the suites exclusion rules. This way, things like external API calls modifications will not initiate a UI automation suite that is not even getting to this logic when running, as it is running with mocks.
An example of such a configuration for one of the suites in our team:
Important to note here is that all deployments we are discussing are Simulator environment deployments, where our automation suites are running. For Production Deployments, all suites are running in all cases, so we make sure no bug escapes our radar to production.
Results
We tested this mechanism for a month and compared it to the month prior to the implementation, and the results we got were awesome. We saw a reduction of 25–30% in the amount of tests runs when comparing the two periods. This directly translates to faster deployments, cash savings, and happier developers. Combine this with an easy plug-and-play per-service implementation, and get all those benefits multiplied by the number of teams in the company.
This mechanism does not take a lot to maintain, is almost bug-free (at least until this writing), is pretty easy to enhance with extra features (as it’s essentially serverless and not a full service), and its ROI is strong positive.
With some creativity, and even if it doesn’t seem obvious at first, there are always ways to improve the existing infrastructure, and we are constantly looking for them.
