How to Safeguard Your Data Connection in Jira?
Table of Contents
- Introduction
- Quick Recap — What are Data Connections in Jira?
- Why we need to safeguard our data connection?
- How we have safeguarded it?
- Our former setup
- The safeguard setup - Conclusion
Introduction
In the fast-paced world of engineering at KPMG-UK, transitions are a constant. Recently, we made a significant shift from Company-managed Jira Service Management (JSM) to Team-managed JSM. This shift was driven by three key factors: Autonomy, Portal Separation, and Regaining Focus. If you’re curious about the intricacies of this transition, we’ve covered it in detail in our previous blog post, “KPMG UK Engineering’s Move to Team-Managed JSM Projects.”
However, with change comes challenges, and one of the hurdles we faced during this transition was safeguarding our data connections in Jira. During this transition, we encountered a few issues with data connections. The major problem was that the data obtained from the source endpoint sometimes became unavailable due to (even brief) outages, causing significant problems with automated JSM requests. As most of our automated requests rely on this data connection value to perform tasks.
In this blog, we focus on how we have safeguarded our data connections by ensuring the data is available 24/7, preventing automation failures for our JSM requests due to unavailable data.
Before we dive into the solution, let’s refresh our memory on what data connections are.
Quick Recap — What are Data Connections in Jira?
Data Connections are a useful tool implemented within Jira Service Management that allow the population of Form fields with live data gathered from API Calls. This applies to radio buttons, checkboxes and select dropdown fields.
What does it mean? Lets visualise!
To know more about the data connection refer our previous blog, “ How KPMG UK Leverage Jira Data Connection.”
Why we need to safeguard our data connection?
Now that we understand what a data connection is and how it can be created, let’s understand why we need to safeguard it.
- Timeout Concerns: We use proxy lambdas to retrieve data from external sources like the internal repositories of GitHub. It’s important to note that while proxy lambdas solve a host of issues like pagination, authentication, and helping when the source API endpoints are present in private services(like VPC), they also come with limitations.
One such limitation we’ve encountered is the 30-second timeout limit of lambdas associated with API endpoints. Currently, the time required to retrieve data from internal repositories can exceed 30 seconds. Consequently, we can’t fetch the data in time, leading to no options in the dropdown field. This, in turn, breaks our automation, as these data connection values are used to perform tasks. - API Service Downtime: Apart from timeouts, another major issue arises when the source API service used to fetch data goes offline or breaks down. This situation leads to no data being provided in the data connection-linked fields, again breaking our automation when these values are not present.
To overcome these problems, we’ve come up with a quick fix to safeguard the data even if the retrieval time is longer or the source API service is down.
How we have safeguarded it?
Before delving into the solution, let’s have a clear understanding of our former setup:
Our former setup
Initially, our data connection setup was structured as follows:
- Multiple Python Files: Each Python file was responsible for fetching specific data from various sources.
- Single Serverless File: This file was used to deploy all the Python files as Lambda functions and to have an API gateway as a trigger for them on AWS.
- Proxy Lambdas: For our use case, we are specifically utilising Proxy Lambdas. This means that when a lambda is triggered by an API call (from the Data Connection) instead of the API gateway handling the response, it is passed straight through to the Lambda function. Once the Lambda runs and a response is received, the API gateway passes it straight back to that initial incoming lambda.
File structure: In the former setup, we used to have multiple Python files responsible for fetching specific data from various sources and a single Serverless file to deploy them in AWS.
The only disadvantage with a single Serverless file is that it used to take a long time to deploy, as it would deploy all the data connections defined in that Serverless file.
The safeguard setup
In our safeguard setup, we’ve introduced two Lambda functions and an S3 bucket to ensure the reliability and resilience of our data connection:
- Data Collection Lambda: This Lambda function is responsible for collecting data from the source. Once the data is collected, it is uploaded to an S3 bucket for storage.
- Data Population Lambda: This Lambda function retrieves the data from the S3 bucket and populates it into the data connection. By directly accessing the data stored in the S3 bucket, this Lambda ensures that the data is always available for use in the data connection.
File structure: In the new file structure, we’ve organized the setup by creating subfolders for each service-specific data connection. Each service now has its own serverless file dedicated to deploying it.
What does this mean? Whenever you create a new data connection, you just need to deploy that service-specific serverless file. This approach helps in saving deployment time, as you no longer need to deploy all the data connections at once.
Creating the Safeguard setup with the Serverless Framework
Here’s how we handled this via code using the Serverless Framework for AWS. For example, let’s consider we want to get the list of Jira projects present in our Jira cloud instance.
1) collect_jira_projects.py :
Start by creating a Python script called collect_jira_projects.py. This script collects the Jira projects in JSON format from the source and uploads them to the S3 bucket.
a. Define an S3 bucket where Jira project data will be uploaded in JSON format:
b. Add a function to fetch the Jira projects from the Jira cloud instance:
c. Add a function to create and upload the json file to that S3 bucket:
d . The handler (main) function:
2) populate_dc_jira_projects.py :
Next, create another Python script called populate_dc_jira_projects.py. This script retrieves the data present in the S3 bucket and populates it into the data connection
a. Define the S3 bucket where it needs to fetch the data:
b. Add a function to get data from the S3 bucket:
c. The handler (main) function to populate the data using the proxy lambdas: Check out our previous blog on proxy lambdas — “How KPMG UK Leverage Jira Data Connections”
3) serverless.yml :
At the end, have a serverless.yml file to allocate the resources in AWS and deploy them
a. Define the environment variables(like S3 bucket name) in the “environment:” block:
b. Define the S3 bucket configurations in the “resources:” block:
c. Define the two lambdas (collect and populate) in the “functions:” block:
After deploying the serverless.yml file (How to deploy it? Follow the Serverless deploying guide), you’ll receive an API endpoint. This endpoint is defined in the “events:” section of the ‘populate-dc-jira-projects’ lambda function. By adding this API endpoint to the Jira data connection settings, you can integrate this data connection into your forms.
Conclusion
To wrap it up, the integration of Jira’s Data Connection feature has turbocharged our JSM operations, elevating it to a whole new level 🚀 . With clever proxy lambda workarounds and robust safeguard setups in place, we’ve turned potential pitfalls into opportunities for innovation.
While Data Connections may not be perfect, their versatility and potential, when paired with the right expertise, make them a game-changer within any JSM instance. Here’s to embracing the future of automation and efficiency with open arms! 🌟
I hope you found this article useful, let me know your thoughts in the comments section.
If you enjoyed reading this blog post, give it a clap 👏, hit the follow button, and subscribe to stay updated with my future articles.
Connect with me on LinkedIn for even more updates: https://www.linkedin.com/in/preetham-dp-063584195/
