Automating Github to Jira Using Python in AWS Lambda with Serverless
This project is intended to provide a template to integrate version control system such as Github with Jira ticket system using AWS lambda with serverless deliver
Codes from GitHub
Objectives:
Create a seamless and automated integration between a version control system, such as GitHub, and the Jira ticket system, utilizing AWS Lambda and a serverless delivery approach. This integration aims to enhance collaboration and streamline development workflows by automatically linking code changes to relevant Jira tickets, ensuring efficient tracking and management of software development tasks.
Tools:
Serverless (SLS)
Serverless architecture is employed in this project to maximize scalability, minimize operational overhead, and optimize cost-efficiency. By leveraging AWS Lambda and serverless computing, we can dynamically allocate computing resources as needed, eliminating the need for manual server provisioning and maintenance. This not only ensures that our integration is highly available and can handle varying workloads but also reduces infrastructure-related expenses. Serverless also allows for event-driven execution, enabling seamless integration with GitHub and Jira, as functions can be triggered in response to specific events, such as code commits or ticket updates. Additionally, serverless promotes rapid development, enabling us to focus on the core functionality of our integration, rather than managing infrastructure, thus accelerating our time-to-market.
AWS Lambda
AWS Lambda is the ideal choice for this integration project due to its serverless nature and event-driven capabilities. By utilizing Lambda functions, we can execute code in response to specific events, such as GitHub commits or Jira ticket updates. This event-driven architecture ensures that our integration stays synchronized and responsive without the need for continuous polling or maintaining a persistent server. Lambda offers automatic scaling, so it can effortlessly handle varying workloads, making it highly cost-effective by only charging for actual execution time. Moreover, Lambda integrates seamlessly with other AWS services and provides robust security and monitoring features, ensuring the reliability and security of our GitHub-Jira integration. Overall, Lambda empowers us to build a scalable, cost-efficient, and highly responsive integration solution while minimizing operational complexity.
Python
Python is chosen as the primary programming language for this integration project due to its versatility, simplicity, and extensive libraries and frameworks available. Python’s readability and ease of use make it an excellent choice for writing and maintaining Lambda functions, allowing for faster development and reduced debugging time. It offers robust support for API interactions, making it well-suited for communicating with GitHub and Jira’s APIs, simplifying the integration process. Additionally, Python has a vibrant developer community, providing access to a wealth of open-source packages and resources, which can significantly expedite the development process and ensure the long-term maintainability of the integration. Overall, Python’s combination of readability, ease of use, and rich ecosystem makes it an ideal choice for building a reliable and efficient GitHub-Jira integration solution.
Prerequisites:
FYI: Ubuntu solution is provided in this project, but Windows and Mac reference will be given
- Install Serverless on Ubuntu
- Configure AWS programmatically to communicate in between Serverless and Lambda
- Create a repo in Github (webhook of the repo needs to configure as needed)
- Create a project in Jira (api token of admin user is required)
Step by step instructions:
Install Serverless on Ubuntu (You may spin an instance in AWS or via Virtual Machine locally such as VMWare)
Install Node.js and npm:
If you don’t already have Node.js and npm installed on your Ubuntu server, you can do so by running the following commands:
sudo apt update
sudo apt install nodejs npmInstall Serverless Framework:
Once you have Node.js and npm installed, you can install the Serverless Framework globally using npm:
npm install -d serverless@3.38.0This command installs Serverless Framework globally so that you can use it from the command line.
@3.38.0 you may replace this version with serverless latest version from here
Verify the Installation:
To ensure that Serverless has been installed correctly, you can check its version:
serverless --version
FYI: Please be mindful that latest version of serverless needs to be installed. Otherwise, lambda function deployed may return with handler missing on module
I spent quite a lot of time on this for troubleshooting, would like to see you save time on this :)
Install AWS CLI:
You can install the AWS CLI on Ubuntu with the following commands:
sudo apt install awsclicheck aws cli
aws --version
Configure AWS programmatically to communicate in between Serverless and Lambda
To configure aws cli with default profile
aws configure
Only below 2 values need to be provided
AWS Access Key ID []:
AWS Secret Access Key []:
Where should you obtain them?
Let us jump into AWS console on IAM
Make sure you have a user with admin user (if you don’t allow admin level of access, you must be granted with all permissions need to interact with CloudFormation for resource creation in AWS)
FYI: You may need to test along the way if you don’t have admin level of access since programmatic access is associated with policies attached the user
Create a repo in Github (webhook of the repo needs to configure as needed)
After creating an account, you may land above page
We may create a repo as shown below
Create a new repo
What you need is a blank repo only
We will configure settings under this repo when needed
Create a project in Jira (api token of admin user is required)
Following the instructions, you may easily create an account
Then click Jira Software to land on ticket page
Now you may create a project
You may want to choose scrum as a type
Make sure you note down the Key since it’s needed later on when we configure values needed
At this moment, we are all set for our project to kick off!
Here is a folder tree you may want to refer to for our serverless project
FYI: Keep in mind, there are caveats why I prefer to do in one way not another, but other alternatives are provided as well
Now let us get our hands dirty with files we need
I will explain every file and folder shown above, but you don’t need to create all of them
First and foremost, we’d better create an independent folder for this project
mkdir serverless-automation-jiraThen we start off files and folders creation inside this folder
cd serverless-automation-jiraIf using visual studio code, you can definitely create these files and folders using UI as shown below under the top level folder
You may also use vim, vi, or nano using command
FYI: It is worth of setting up Visual Studio Code locally as a DevOp or SRE as per say :)
Our core files are here
serverless.yml
service: ${env:SERVICE_NAME}
provider:
name: ${env:AWS_PROVIDER_NAME}
runtime: ${env:AWS_RUNTIME}
stage: dev # ${env:STAGE}
region: ${env:AWS_REGION}
configValidationMode: off # Disable provider validation warning
custom:
# Add the variablesResolutionMode option here
variablesResolutionMode: 20210326
functions:
GithubJiraAutomation:
handler: ${env:LAMBDA_HANDLER}
events:
- http:
path: ${env:API_PATH}
method: ${env:API_METHOD}
package:
artifact: ${env:LAMBDA_FUNCTION}
environment:
github_secret: ${env:github_secret_value}
jira_api_endpoint: ${env:jira_api_endpoint_value}
jira_api_token: ${env:jira_api_token_value}
project_key: ${env:project_key_value}
jira_user: ${env:jira_username›_value}
layers:
jira:
path: ${env:JIRA_LAYER_PATH}
compatibleRuntimes:
- ${env:LAYER_RUN_TIME}
plugins:
- serverless-python-requirements
package:
individually: true
exclude:
- ${env:EXCLUDE_PATTERNS}
include:
- ${env:INCLUDE_PATTERNS}
useDotenv: trueVariables are used mainly unless there are errors returned like
stage: dev # ${env:STAGE}Provider section is to connect with AWS
FYI: Here I’d like to highlight the importance of runtime value. If you choose runtime as python3.8 for instance, your layers must match with this lambda function’s runtime. In some cases, there are dependencies that can be only applied under certain runtime
When testing lambda function, below error returned for PIL
ImportError: cannot import name _imaging
To better understand how to handle errors related, we may check PIL package page
From above page, we learn we may do so in our lambda_function.py file
from PIL import Image
from PIL.Image import core as _imagingHere’s another dependency related issue resolved
Serverless-python-requirements — The “path” argument must be of type string
package:
individually: trueCustom section is to provide certain option needed in the file
Functions section is to create functions (remember, you may create more than one function in this section with same pattern)
FYI: Here I’d like to explain a bit about usage of handler and artifact
Handler is to use lambda_function.py, which is a default option to generate the code in lambda function (this is a value that must be provided)
Artifact is like uploading zipped file to generate code in lambda function (this is an optional choice, but it turns out to be a better alternative in this project since we need to pack everything in. Otherwise, we may need to add multiple layers for dependencies)
Environment is to create environment variables under configuration section in lambda function (I add them here to better manage them rather than manual creation)
FYI: We need to consider any way possible to avoid any UI involvement since it may cause more human related errors. Also, it’s better off to manage all variables in one place to avoid any confusions
If both handler and artifact are used, artifact is applied as I tested since handler must be provided in serverless.yml file
Layers section is not necessary in this project as I were able to pack everything into artifact at the end of the day. But it’s worth of understanding this option to add dependencies when they can not be packed in zipped file or you prefer not to pack them in zipped file
Plugins section to include plugins that might be needed
Package section to include packages need to be included and excluded
useDotenv: true is used to load environment variables from a
.envfile into your Serverless service configuration, making it easier to manage configuration settings and secrets in a development-friendly and secure manner.
FYI: Using variables in a single place is also a good practice to maintain in the long run
lambda_function.py (this is not the code applied in lambda function, but required to provide in serverless.yml file)
FYI: Please note this file is not used in any way for this project. You can actually replace it with a template python file to pass python file check by serverless.yml file
import hashlib
import hmac
import json
import os
import requests
import re
from jira import JIRA
def verify_github_webhook(event, context):
# Replace 'YOUR_GITHUB_SECRET' with your actual GitHub webhook secret
github_secret = os.environ.get('github_secret')
if github_secret is None:
return {
'statusCode': 500,
'body': 'GitHub secret is not configured.'
}
# Get the request headers and payload
print(json.dumps(event, indent=2)) # Print the entire event for inspection
# Access individual headers from the 'headers' object
headers = event.get('headers', {})
print("Headers:", headers)
body = event.get('body', {})
print("Body:", body)
# Check if the 'x-Hub-Signature' header is present
if 'X-Hub-Signature' not in headers:
return {
'statusCode': 400,
'body': 'X-Hub-Signature header is missing.'
}
# Retrieve the signature from the headers
provided_signature = headers['X-Hub-Signature']
# Calculate the expected signature using the secret and payload
expected_signature = 'sha1=' + hmac.new(
github_secret.encode('utf-8'),
body.encode('utf-8'), # Use 'body' instead of 'payload'
hashlib.sha1
).hexdigest()
# Compare the provided signature with the expected signature
if not hmac.compare_digest(provided_signature, expected_signature):
return {
'statusCode': 403,
'body': 'Signature verification failed.'
}
return {
'statusCode': 200,
'body': 'GitHub webhook connection verified.'
}
def create_jira_ticket(event, context):
# Replace 'YOUR_JIRA_API_ENDPOINT', 'YOUR_JIRA_USERNAME', and 'YOUR_JIRA_PASSWORD' with your Jira details
jira_api_endpoint = os.environ.get('jira_api_endpoint')
jira_username = os.environ.get('jira_username')
jira_password = os.environ.get('jira_password')
project_key = os.environ.get('project_key')
try:
# Initialize a Jira object with your API credentials and server URL
jira = JIRA(server=jira_api_endpoint, basic_auth=(jira_username, jira_password))
# Parse the GitHub webhook payload JSON
print(json.dumps(event, indent=2))
github_payload = json.loads(event.get('body', '{}')) # Parse 'body' as JSON
# Extract relevant information from the GitHub payload
# You may need to adjust this based on your specific use case
repo_name = github_payload.get('repository', {}).get('name')
event_type = github_payload.get('action')
issue_title = github_payload.get('issue', {}).get('title')
issue_url = github_payload.get('issue', {}).get('html_url')
# Create a Jira ticket based on the GitHub update
jira_data = {
'project': {'key': project_key}, # Replace with your Jira project key
'summary': f'GitHub Update for {repo_name} - {event_type}: {issue_title}',
'description': f'GitHub Issue URL: {issue_url}',
'issuetype': {'name': 'Task'} # You can choose the appropriate issue type
}
# Make a POST request to create the Jira ticket
issue = jira.create_issue(**jira_data)
return {
'statusCode': 200,
'body': f'Jira ticket created successfully. Issue Key: {issue.key}'
}
except Exception as e:
return {
'statusCode': 500,
'body': f'Error creating Jira ticket: {str(e)}'
}
def lambda_handler(event, context):
# Call both functions
result1 = verify_github_webhook(event, context)
result2 = create_jira_ticket(event, context)
# You can choose how to handle the results or combine them as needed
combined_result = {
'verify_github_webhook_result': result1,
'create_jira_ticket_result': result2
}
return {
'statusCode': 200,
'body': json.dumps(combined_result),
'headers': {
'Content-Type': 'application/json'
}
}requirements.txt
aws-lambda-powertools==2.32.0
certifi==2023.11.17
charset-normalizer==3.3.2
defusedxml==0.7.1
idna==3.6
jira==3.6.0
oauthlib==3.2.2
packaging==23.2
Pillow==10.2.0
pip==21.2.4
pkg-resources==0.0.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
setuptools==58.0.4
typing-extensions==4.9.0
urllib3==0.0.0FYI: requirements.txt file in Python typically lists the dependencies (external libraries or packages) that your project relies on, along with their versions
node_modules, package-lock.json, package.json, .serverless files and folders are created when deploying serverless
FYI: If you encounter some errors, you may delete these folders or files to rerun sls apply to see if errors may be resolved
.env
# Replace these placeholders with your actual values
# Service-related variables
SERVICE_NAME=github-jira-automation
LAMBDA_FUNCTION_NAME=GithubJiraAutomation
# AWS Provider-related variables
AWS_PROVIDER_NAME=aws
AWS_RUNTIME=python3.8
STAGE=dev
AWS_REGION=us-east-1
# Function-related variables
LAMBDA_FUNCTION=./test_final.zip
LAMBDA_HANDLER=lambda_function.lambda_handler
API_PATH=/webhook-handler
API_METHOD=ANY
#github_secret=github_secret
github_secret_value=********
#jira_api_endpoint=jira_api_endpoint
jira_api_endpoint_value=********
#jira_api_token=jira_api_token
jira_api_token_value=*******
#jira_user=jira_user
jira_username_value=*******
#project_key=project_key
project_key_value=JR
# Layer-related variables
JIRA_LAYER_PATH=./jira-layer
LAYER_RUN_TIME=python3.8
# Resources-related variables
LAYER_PERMISSION_TYPE=AWS::Lambda::LayerVersionPermission
LAYER_PERMISSION_ACTION=lambda:GetLayerVersion
LAYER_PERMISSION_PRINCIPAL=lambda.amazonaws.com
LAMBDA_LOG_GROUP=AWS::Logs::LogGroup
LAMBDA_LAYER_VERSION_ARN=*
# Plugins-related variables
SERVERLESS_PLUGIN=serverless-python-requirements
# Package-related variables
EXCLUDE_PATTERNS='node_modules/**'
INCLUDE_PATTERNS='./lambda_function.zip'FYI: It’s literally just a variable file to assign values to the variables needed in serverless.yml file
The last file is the zipped file using to apply codes in lambda with dependencies
The folder structure of the zipped file before zipping it
FYI:
One thing that needs your attention is that lambda_function.py file must be at the root level in the zipped file
Using UI, you may do as shown below
If using command, you may do as shown below
cd lambda_functionThen, we may zip everything in this folder into a file
zip -r test_final.zip ./*switch into the folder with all dependencies and lambda_function.py file
Voila, we are all set now!
Let us deploy our project
Make sure you double check in .env file
LAMBDA_FUNCTION=./test_final.zipThis value controls over which zipped file we use to deploy
Now deploying the project using sls
sls deploy
Upon successful deployment, you may see below output
This project takes about 100 secs to deploy
Now, let us check our deployment in AWS console
Serverless creates a CloudFormation, which is the AWS native Iac tool
You may check different tabs under CloudFormation page
Resources: all resources created by this deployment
Outputs: values of services that may be considered
We may first test our lambda function as shown below
Now, when we check our jira ticket, we may see a ticket without expected values
FYI: Don’t be freaking out when 400 code returns and values expected not shown. We’re missing one step, which is to configure our Github Webhook
Just to briefly explain the infrastructure here
We take advantage of AWS API Gateway to listen from GitHub Webhook, which allows the API Gateway to collect data and records from GitHub, then it helps trigger Lambda function
The beauty of this infrastructure is that
The cost is lowered to the minimum, which we call pay as you go.
As shown from above lambda execution screenshot
Billed duration: 1535 ms
Which means we only pay for 1.535 second
vs running an instance
With Lambda, infrastructure is managed by AWS, so overhead for infrastructure building is also saved
Now we will configure out Webhook in Github
Under repo page, select settings, then on the left lane, choose webhooks
Copy the highlighted api endpoint of API Gateway
Then paste it into Payload URL in Webhook
Content Type: application/json
SSL verification:Enable SSL verification
Which events would you like to trigger this webhook? Send me everything.
Active — We will deliver event details when this hook is triggered.
Update webhook after above values are checked
Now final testing
We will only uncheck starred as shown below
We move to webhook now to check recent deliveries — payload
{"verify_github_webhook_result": {"statusCode": 200, "body": "GitHub webhook connection verified."}, "create_jira_ticket_result": {"statusCode": 200, "body": "{\n \"id\": \"10051\",\n \"key\": \"JR-20\",\n \"self\": \"https://zhaofeng8711.atlassian.net/rest/api/3/issue/10051\"\n}"}}
FYI: I configured jira ticket based on my understanding of payload shown, you can definitely tweak the ticket as you’d wish
check ticket creation function in lambda_function.py file
def create_jira_ticket(event, context):
# Jira API URL
url = os.environ.get('jira_api_endpoint')
# API token and user email from environment variables
api_token = os.environ.get('jira_api_token')
user_email = os.environ.get('jira_user')
auth = HTTPBasicAuth(user_email, api_token)
headers = {
"Accept": "application/json",
"Content-Type": "application/json"
}
# Parse the GitHub webhook payload JSON
github_payload = json.loads(event.get('body', '{}')) # Parse 'body' as JSON
# Extract relevant information from the GitHub payload
repo_name = github_payload.get('repository', {}).get('name')
created_at = github_payload.get('repository', {}).get('created_at')
issue_url = github_payload.get('repository', {}).get('html_url')
action = github_payload.get('action', {})
# Create the Jira ticket payload
payload = json.dumps({
"fields": {
"description": {
"content": [
{
"content": [
{
"text": f"GitHub Issue URL: {issue_url}, {action} at {created_at}",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
},
"project": {
"key": "JR" # Adjust the Jira project key as needed
},
"issuetype": {
"name": "Task"
},
"summary": f"GitHub Issue in {repo_name}: something is {action}",
},
"update": {}
})
response = requests.post(url, data=payload, headers=headers, auth=auth)
return {
'statusCode': 200,
'body': json.dumps(json.loads(response.text), sort_keys=True, indent=4, separators=(",", ": "))
}You may assign values here
repo_name = github_payload.get('repository', {}).get('name')
created_at = github_payload.get('repository', {}).get('created_at')
issue_url = github_payload.get('repository', {}).get('html_url')
action = github_payload.get('action', {})Then you may create a ticket as you prefer in payload
payload = json.dumps({
"fields": {
"description": {
"content": [
{
"content": [
{
"text": f"GitHub Issue URL: {issue_url}, {action} at {created_at}",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
},
"project": {
"key": "JR" # Adjust the Jira project key as needed
},
"issuetype": {
"name": "Task"
},
"summary": f"GitHub Issue in {repo_name}: something is {action}",
},
"update": {}
})OK, let us now check in Jira
The first ticket is our testing ticket without values assigned from Lambda Function testing
The second ticket is our intended ticket
The payload expected is all here
Now we may wrap up our project
Since it’s testing project, don’t forget to churn the full project with
sls remove
After 27 seconds, we deleted all resources created via CloudFormation
Extra points to add:
Here I’d like to point out a few things that might be worth of attention
To troubleshoot in case of an issue facing the lambda function, please get to the CloudWatch log group configured for this project
For easy access, select view CloudWatch logs under Lambda Function page
Log streams with time lines are provided in details
Sample log page
For permissions needed in Lambda Function, we have one resource based permission to allow API Gateway to trigger the Lambda Function
And 3 other CloudWatch Logs related default permissons
FYI: You may tweak permissions as you prefer based on your needs, but rule of thumb is least privelege should be granted for the best security practice in AWS
Don’t forget our pack files are all restored in S3 bucket created, you have access to all of them there for your reference in the future
One more thing is that whenever updating anything in Lambda Function when testing your codes, make sure you publish a new version and update the codes. Since new Lambda function will not be deployed without doing so
FYI: Please don’t update like this for production environment, since there will values you update missing in .env file in serverless folder. In that way, confusions might be caused in the future
