Managing deployment cycles for hundreds of repositories
Hi people! In this article, I will tell the story of deployment cycles and how we are handling them.
Let's start… From the beginning.
How are we deploying the repositories?
There is more than one way to deploy them to production. I will share a few examples.
- GitHub Action
- AWS Codebuild
- AWS Codepipeline
But what is the common ability of these tools? Yes, you already knew it, triggering by WEBHOOK!
These tools are getting triggered from GitHub Webhooks. There is a lot of webhooks on the GitHub side for almost all event, but we are using the specific one, which is the merge event. I mentioned some tools earlier, these tools are listening to the merge event for the master branch. So, when we sync the master with develop branch they are working. Taking code from GitHub, building, and sending it to the next step. The next step could be a Lambda, EC2, EKS(Kubernetes), or Elastic Beanstalk on AWS. After this pipeline, the source code is going to production. That's all! We finished.
It's a joke, our story begins here.
The master of the tickets: JIRA
Jira is a software application used for issue tracking and project management. We are using JIRA to develop our projects step by step without any complexity in my company. The tickets open, develop, test, and deploy via JIRA steps.
Yes, you heard me, I said we are deploying the projects via JIRA steps but I did not mean that. We are using the steps (which will be mentioned as status in the next chapters) to track our tickets.
The Code Base: GitHub
GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere.
In Insider, we have active repositories of around 200 only for production services.
Also, we have over 600 pull requests to handle every month.
We need to mention here something. We are opening the branches on GitHub via Jira Task ID. For example, if the developer is developing a task with id SD-12345, the GitHub branch should be feature/SD-12345 or SD-12345. We are easily able to match up Jira Issue and GitHub Pull Requests. There can be more than one same branch name for the different repositories. I mean if the developer is writing code to two different repositories for the same task, the developer should open the branches with the same name.
The Management of Deployments: Arcanine
Now, I will tell the story of how we are managing the pull requests via Jira tickets every day.
Arcanine is a deployment management bot to handle the Jira Issues and GitHub pull requests by merging both sides together. Arcanine was written with Nodejs and Express.
First step: Collecting Data
The Arcanine bot has two main endpoints.
One of them is /jira-webhook endpoint which is listening to the Jira tasks and updates with JIRA Hooks. The hooks send the specific tasks’ data to the Arcanine endpoint. We are using the following JQL to filter the tasks on the Jira side.
So we are not sending all data to the Arcanine to decrease the requests count to the bot. Example data for an issue from Jira.
Another endpoint is the /github-webhook. GitHub endpoint listening to the Pull Requests via GitHub Webhook. Again we are not sending all data to the bot, we only sending Pull Requests events to decrease the requests count. Example GitHub data for an issue:
Arcanine is always listening that the two endpoints and saving data to its own database using Jira issue id as the primary key.
Second step: Releases
Now we have collected data from Jira and GitHub. We did the development for a Jira task. After the all development steps, the developer should move the Jira issue to the Awaiting Release column on Jira. This is the special column for the Arcanine bot because the bot understands the task should be merged on the next release when the developers send their tasks to Awaiting Release.
Arcanine is creating a new Release every day at 2.00 PM. Release means that the bot checks the Jira issues and sends confirmation messages via Slack to developers and QAs if the issue is in the Awaiting Release column.
The developer and QA should give approval to the task by Slack until 8.00 PM every day on weekdays. The bot after getting confirmation from the task’s developer and QA, marks the task as approved.
If a new task comes to the Awaiting Release column between these hours, the bot immediately sends a confirmation message via Slack to the developer and QA for this.
At 8.00 PM the bot is sending the common channel the list of the approved tasks. These tasks’ pull requests will merge with the develop branches of the repositories the next day (weekdays).
Third step: Merge Process
At 7.00 AM in the morning, the bot is getting the last (not merged) release tasks’ pull requests and trying to merge to the base branches. The bot is merging the pull requests if everything is good and the pull requests are mergeable. It is checking whether the task’s pull requests are mergeable or not before merging them to the base branch for all tasks. Sometimes the tasks have more than one pull request and we do not want to merge the task’s pull requests if one of them is not mergeable.
Bot sending the task list message to the common channel to inform developers and QA’s to start the pre-prod stage.
Also, the bot is sending a request to Jenkins to create a new pre-prod environment.
Final!
I know something is missing, what is happening in the pre-prod stage? I will share it with another story.
Hope, you enjoyed reading it! 🤞
Let me know if you need detailed information about the structures of the deployment.
Open for your comments and suggestions in the comments 📝.
And feel free to contact me on LinkedIn as well.
