Don’t Repeat Job-self!
By Francisco Javier Garcia Orduna & David Campos Vall
At King, we use Jenkins extensively to build our games and execute our tests. All of King’s development departments use this tool as there is a high-throughput of screenings, which generate large volumes of diverse data. This requires a flexible approach when building integrated, scalable and robust computational workflows for building, as maintaining our games is challenging.
Jenkins You Say?
Jenkins is an open source implementation of a Continuous Integration (CI) server written in Java that can be used as a self-hosted option automating the build cycle for any project. It is a well-known, open-source continuous integration server. The CI is an established practice in software engineering and, especially at King, one that supports the development of complex software programs. A continuous integration server is designed to automatically or manually trigger complex workflows to build, test, and release software components. Normally these platforms also provide process monitoring, testing, and validation tools. Jenkins offers a simple way to create a CI for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks.
While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does offer a faster and more robust way to integrate your entire chain of build, test, and deployment tools that you can easily build yourself.
How It Works at King
In the Build, Test & Release area of Jenkins there are two main workflows, “Merge to Meta” and “Hotfix”, to build, test and fix all our libraries and our Meta Release, which are the most common sets of shared libraries used in all King games.
For example, each of these workflows could possibly launch at least 83 building jobs and this amount of jobs might potentially increase in the future due to the addition of new modules and features added every quarter. Therefore, they must be tested in order to deliver a good quality product. Now, you may not know how many times the same Jenkins job is repeated with exactly the same parameters within King’s Jenkins servers. For the moment, let’s just say it’s substantial!
Jenkins is a great tool but it relies heavily on the surrounding environment, like the network, Jenkins agents and their configuration aspects. Sometimes, it might be the case that a job fails due to something not related to the job itself.
Some examples of this could be a failing connection in a GitHub repository, a timeout in the pending jobs queue or a disk space error in the Jenkins agent. One single job failure can ruin all the work done during the workflow execution.
So, what can we do to avoid the redundant execution of something that we know worked before?
Jenkins Pipelines? I’m Afraid They Wouldn’t Work…
The Pipeline Plugin allows the creation of checkpoints in the pipeline, in order to resume execution if something goes wrong. This capability works perfectly when there is a sequential execution of phases and some checkpoints are set up between them. However, we want to run everything in parallel in order to save compilation time.
The Jenkins Pipeline Plugin can’t manage a “partial” checkpoint for the cases where there is a building phase with more than one building job, maybe dozens, all of them launched in parallel. In other words, the checkpoint might resume the status of the whole building phase but not a part of it.
So, that’s why this approach doesn’t work to solve our problem because we want to avoid every single redundant execution, especially when it is was executed in parallel.
DIY: Our Jenkins Plugin!
Our initial search in the Jenkins Plugins page didn’t show any plugins that would solve the problem, so we had to research it from scratch.
We wanted the plugin to ask for the status of a previously executed job at the beginning of the job execution: if it was a success, then it would just interrupt it and return a “SUCCESS” result, otherwise, it would keep on executing the job as expected. After execution, the ideal plugin would save the result of the execution, just in case the job is executed again in the future.
With this requirement in mind, we eventually found there was a type of plugin implementation that would suit our needs perfectly called a Wrapper Plugin. This type of implementation would allow some actions at the beginning and at the end of a job execution. It literally “wraps” the job execution.
Now, in order to save a job result, we needed to accomplish two things: a way to identify if two job execution results are the same and a place to store this information.
To address the first goal, our plugin generates a hashed value that allows the identification of not only the job’s name but also the Jenkins master and its most relevant parameters (names and values). So, if you execute the same job twice with different parameters, a different identifier will be generated.
For the second goal, the plugin uses an external database to store the relationship between the job’s hashes and its output. As we don’t want to save this information forever, we went for a typical key-value cache approach provided by an external service, accessible using HTTP protocol.
Ask for the previous job result at the beginning of the job execution
Save the job result at the end of the job execution
We called the resulting plugin Results Cache. Cool, eh?
First Impressions in the Real World
After some time using the plugin in the Meta Workflows, we noticed a considerable decrease in resource usage (IT and human) when something went wrong.
Cached jobs only take a few seconds
Now users can relaunch the whole process themselves without having to wait for every single job to be executed; rather, they just wait for the ones that failed. The use of Jenkins agents in these cases is much lower than before, and as a result, the Build, Test & Release area representatives receive fewer support requests.
Moreover, using this new plugin allowed us to use the Naginator Plugin, which allows you to automatically reschedule a build after a build failure. Taking into account that sometimes a job fails due to external conditions, the combination of both plugins allow users to forget about their job because it will be retried automatically.
Even though the plugin is working fine in the Meta Workflows scenario it still needs further improvements, for example, if we wanted to fix some incompatibilities with other Jenkins Wrapper Plugins. This is why open source projects are very important for tech companies to build collaboration externally while solving software problems.
In the future we hope the main idea for this project is that it will be refined by King engineers and the open source community.
In our experience at King, using this Jenkins plugin as well as the CI for software development has been extremely responsive for fixing software issues and providing sustainable improvements. The advantages of Jenkins include an easy setup and maintenance compared with some of the other tools. Jenkins has simple and effective file-based systems for data and results in management without the effort of maintaining a separate database system. This also offers flexibility at early stages of development where requirements are still rapidly evolving. Changing task parameters and the structure of the workflows is straightforward without complex cascading effects. Jenkins also provides features for provenance tracking and collaboration. Build histories and logs provide a detailed record for tracking and reproducing analytical workflows and for troubleshooting any issues that may arise.
From a systems development viewpoint, Jenkins CI has enabled us to rapidly analyse requirements and prototype solutions at the beginning and at the end of a job execution. This combined with our Jenkins plugin solution offers a solid problem-solving tool for King that enables robust building and testing of games.
We managed to achieve our main goal and we hope you’ll enjoy our Jenkins plugin as much as we enjoyed creating it.
Should you have any questions or feature requests, don’t hesitate to reach out to us!
Checkout King Jobs to see our latest opportunities.