Improving processes and development flow with automation, War Dragons case study

By Juan Silva

Abstract

For “War Dragons,” we have processes in place for certain tasks, like adding a feature to the game or making changes to server code. These processes are in place to help with the development flow and avoid issues that we faced in the past. Historically, when these processes weren’t followed, we ran into a set of predictable problems.

Because of these reasons, we decided to improve the workflow and automate certain aspects. We started with a two-layer solution and a system that integrates three different services. The result was a more stable and nicer development flow with less recurring issues and better feedback to developers and contributors to the project.

Context

“War Dragons” uses, among others, three different services for development. They include GitHub for version control, JIRA for project management, and Jenkins for continuous integration. On GitHub, we use a feature called pull request, which is a peer review step before applying changes to the main development branch.

We started with an existing project as base called DragonsBot, a basic GitHub integration tool. The main functionality that we had with it was to run two GitHub checks on pull requests and, only after the verifications were passed, we allowed a pull request to be merged to the main branch.

Process

The “War Dragons” development flow looks something like this:

In general, the coding process starts with a feature ticket in JIRA, a branch in GitHub, and two Jenkins jobs that build the game for Android and iOS, respectively. The ticket and the branch is created manually while the Jenkins job creation is semi-automated, either created when opening a pull request or created via our internal website manually.

When the feature is complete or stable enough, QA starts testing the feature for potential bugs, edge cases, and other issues. When QA is testing the feature, the status of the feature ticket changes from Development to QA.

After QA testing is complete and satisfactory, we move the ticket to Code Review where we can start reviewing the changes via pull requests. During this, we must check the code changes and verify that the Android and iOS builds are still successful after making changes to the code based on feedback. We also need to verify other aspects of the changes such as standard line endings for CSV files, proper image format, missing localization items, properly resolved conflicts, etc.

After code review is approved, QA gives a final pass. If everything is good, then we move the status of the JIRA ticket from Code Review to Merge, indicating to the owner of the feature that we can safely merge the changes into the main branch.

This flow involves different internal systems with no interaction between each other. Due to that, we ended up with multiple issues in terms of not following the process as we should, missing checks, and breaking specific platform builds without knowing.

DragonsBot Improvements

With DragonsBot as the base project, we started building new functionality to help with the most common pain points while enforcing different processes. The first step was to refactor the project to make it more flexible and extensible so we could introduce new checks or flows easily. We decided to implement each task as its own class, which allowed us to easily choose what task to run given a certain context. For example, if the pull request was a bug ticket, we chose one flow. And if the pull request was a feature, then we used a different flow. In a high level, the end result looked like this:

We also needed to communicate with all of our services to best coordinate and organize the flows that we needed. For that, we implemented some APIs from each service, allowing us to get information and interact with them.

With these integrations, DragonsBot became a coordinator and orchestrator. So, we could then manage the workflow between JIRA, GitHub, and Jenkins.

Automation

With the changes made to DragonsBot and the pre-commit hook, we managed to automate the majority of this process.

For starters, we added GitHub checks that indicate the status of Android and iOS builds on Jenkins. This allowed us to know that at least the changes did not break any builds, because breaking a build involves missing an entire day of QA plus with other issues.

We added several other game-specific checks that were fairly easy to automate while also adding the same checks in the pre-commit hook.

These checks were very specific and highly dependent on some system-level applications. So, we decided to run these as a Jenkins job that then communicates the results to DragonsBot, which then analyzes them and sends them to GitHub.

To further adhere with the development process, we then added another check that reflects the state of the feature ticket in a way that only when the ticket is ready to merge, the check will succeed. Every time the ticket is updated, the update is communicated to DragonsBot from JIRA via WebHook. Then, this update is processed by DragonsBot, which then sends the right information to GitHub to update the check. This functionality allowed us to always have an up-to-date status.

To complement this part of the process, we made DragonsBot to post a message in the corresponding ticket when the pull request is approved, so the individual responsible for QA knows when the ticket can be moved to merge status.

When assigning a pull request to another engineer, GitHub sends email notifications to remind and notify people about it, but that would often get lost in a sea of other emails. So, we also decided to fix that by creating a JIRA ticket that is assigned to the reviewer. That ticket has all the information needed to review the pull request and is also added as part of the sprint, which makes it visible right away.

Results

One of the biggest benefits that we saw was related to time management. It was very common to see issues related to the misuse of different line ending characters and other file format problems. Every week or so, an engineer and a designer would need to triage what was happening with the designer’s setup, and they’d usually spend several hours figuring things out. After we implemented the pre-commit checks and GitHub checks, the issues were completely resolved.

The automatic generation of code review JIRA tickets was particularly useful since it increased visibility dramatically. This allowed us to plan ahead and consider the engineering time involved in the review process, which was something that we were not tracking nor considering before.

Conclusion

Overall, we increased the productivity and performance of the team. We improved the stability of the game and our workflows by effectively integrating and coordinating the three main services that we use for development.

We removed the majority of the human intervention during the development process. We prevented many issues from being merged to master and even being committed on each branch thanks to the GitHub checks and pre-commit hook checks.

Systems like this serve as a good foundation for more solid and resilient teams. It will definitely allow us to be more stable during member changes, accomplish faster onboarding, and develop a better game for our players.