Self-Service Releases
Integrating Slack, Service Now, Jenkins and Github
What is Change Advisory Board (CAB) ?
Historically development team, QA team and IT team will work together to build code, test it and deploy to non-prod environments. Before deploying code to Production, the team would have to represent their change to a Change Advisory Board (CAB). The CAB would meet at regular intervals and may consist of senior management executives, compliance and governance department officials. The team requesting the change would present the change and the CAB approvers would ask typical questions like is the change tested?, is there a down time?, has the relevant parties approved the change for the deployment window?, what is your rollback mechanisms? etc.… Sounds pretty grueling and feel like one more redundant paper work.. Not really..
The CAB serves three purposes:
Firstly, it provides a central platform to communicate and provide visibility to all the changes that will be taking place in various part of the organization, for senior executives, business owners, support organizations and other upstream or downstream development teams.
Secondly, it acts as a gate to ensure all the necessary testing is done and all relevant parties are in agreement for a release.
Thirdly, it provides a process to meet stringent compliance frameworks like SOX, PCI etc..
Is CAB relevant ?
The CAB processes are very relevant when you have a monolithic application that has several pages of pre deployment, post deployment and deployment steps, the risk is significantly higher or you have a change that impacts a lot of different function. CAB also provides proper documentation and approval process framework to provide evidence during an audit.
Cloud implementations have accelerated environment buildouts. A lot of organizations are adopting micro services architecture. Automation tooling like cloud formation, terraform, jenkins, spinnaker and various CI/CD tools have reduced the no of steps to deploy software and automated scripts do very complex tasks at a click of a button. The micro services sub divide a larger application into very small pieces, hence reducing the overall business risk.
Larger organizations still have not moved from the legacy CAB approval process in fear of being in violation of a compliance, and the smaller new companies look at CAB as redundant process that unnecessarily slows everyone down and dont have a CAB or dont follow any process and deploy on the fly.
As software development has significantly evolved so does CAB needs to evolve.
The Middle Ground Solution
As we have smaller automated deployments, we need to build on it to automate some functions of CAB to modernize the process to get compliance, documentation, notification and gating without slowing the dev teams and adding any risk to the organization.
The Architecture:
Part I: Creating the Change record:
In Figure 1 below we describe architecture to use a lambda function to create a Change Record in service now from slack.
- We create a slackbot and configure the slash command. The slash command makes a POST call to the API Gateway. The payload has the Jira project name, start and end time for CR, and the names of the approvers
A sample slash command may look like /create_chg_record <project_name> <jira_label> <start_time> <end_time> <PO_approver> <QA_approver><CM_approver>
eg: /create_chg_record “acme_project release_22” “9/23/2019/06:30” “9/23/2019/09:30” “John PO” “Jane QA” “Rob CM”
2. The POST call from slack triggers a lambda function that parses the payload for <project_name>and <jira_label> and makes a GET call to Jira using JQL and return a list with Jira ticket no and summary to the Lambda function.
3. The lambda function then takes the list of JIRA ticket no and summary and with start_time end_time and list of approvers make a POST call to Service Now to create a Change record. The Service now returns the Change no CHG01234, which is posted back to slack as a confirmation.
PartII: Self-Service Release via slack
In Figure 2 below we describe architecture to use a lambda function to trigger a jenkins job from slack.
- We add a slash command /Deploy to our exiting slackbot. The slash command makes a POST call to the API Gateway. The payload has the Service Now Change record no.
A sample slash command may look like /Deploy CHG01234
2. The POST call from slack triggers a lambda function that parses the change record no and makes a GET call to service now, retrieves the change details and the lambda function runs following validation checks
a) Is the Change Record in approved state
b) Is the current time in the approved change window (between start time and end time of CR )
3. On successfully passing the above checks, the lambda function trigger jenkins job for that app that deploys the code from github branch to EKS or EC2 infrastructure.
4. The jenkins job posts the success or failure message back to slack
Enhancement: To meet SOX or PCI or any other stringent compliance framework we can implement additional check that will make it a three way verification system between JIRA, github and service now. The developers can be enforced to add JIRA ticket no as part of git commit comments. Later at time of code deployment the JIRA ticket nos can be retrieved from service now change record and compared with the git comments. It will be a very stringent system and a bit difficult to implement. It will ensure very high degree of compliance as the code that is being deployed will be compared to what is documented in Change record and the change record will have only items that are labeled in JIRA.
The above automation solves the 3 purposes of CAB:
Communication: Slack keeps a nice stream of start and end deployment times
Gate: Service Now change window and approvers acts as a gate
Compliance: JIRA, Service Now and Slack automates the recordkeeping
Conclusion:
CAB is not redundant its just needs to be revived. With automation and several tools available we can integrate Jenkins, git, service now and slack using AWS api gateway and AWS lambda to implement a solution that simplifies creating change records via slack and makes deployments via slack without compromising Change Management process.
The integration to validate Change record status and deployment window in service now before triggering the jenkins job enables the gating mechanism for Prod changes, and ensures approvals from relevant authorities are procured before the release/deployment. If we decentralize the approvers to relevant product owners, QA leads and management approvers instead of irrelevant CAB members the process becomes less bureaucratic and more agile.
Further, slack notifications in appropriate channels provide a good communication medium so that everyone is alerted of an on-going change or support teams like NOC, Prod Support are able to trace down an alert or an issue to a recently occurred deployment.
Finally, we can extend this framework for stringent compliance frameworks by implementing a three way verification system between service now, github and JIRA.
