Transfer Dashboard Data from Grafana to GitHub Using Python

Published in

Globant

5 min readApr 29, 2024

In this article, we will learn how to download Grafana dashboard data to a local folder and upload it to the “GitHub” repository in two ways. Grafana is a multi-platform open-source analytics and interactive visualization web application that allows users to see their data via charts and graphs that are unified into one dashboard or multiple dashboards for easier interpretation and understanding.

Why do we need to store the Grafana data?

The main limitation of Grafana is that it does not allow you to back up dashboard data that is older than 30 days. We may encounter a situation where our current dashboard is not working properly, and we need to restore the old functional dashboard data; in this case, we should have its backup saved or stored somewhere.

The goals of Grafana dashboards are:

Centralize data: They serve as a centralized hub for integrating data from several sources, allowing users to easily access and analyze information.
Visualize data: They allow users to convert complex datasets into easily understood visual representations such as charts, graphs, and tables, so promoting insights and decision-making.
Real-time monitoring: They enable real-time monitoring of critical systems and services, offering a comprehensive view of performance indicators and facilitating issue detection and resolution.
Performance analysis: They are useful for examining the performance of apps, servers, and networks, allowing users to find bottlenecks, latency, and other performance trends.
Troubleshooting: It provides a comprehensive picture of system behavior, making it easier to diagnose issues such as resource utilization, failures, and anomalies.
Historical Analysis: Users can use these dashboards to analyze previous data trends, allowing them to make data-driven decisions and forecast future behavior.
Custom Reporting: They allow you to build customized reports targeted to specific business needs, and they may be further enhanced with tools like Skedler.

It also supports multiple panels in a single grid. The panels interact with configured data sources, such as AWS CloudWatch, Microsoft SQL Server, Prometheus, MySQL, and many others.

You might be curious about how to store your data, and this article walks you through the process.

Implementation

There are two ways to implement transferring Grafana’s dashboard data to GitHub using Python:

Solution #1: Download the Grafana dashboard data in JSON file format and upload them to a Git repository using the response header.
Solution #2: Download the Grafana dashboard data in JSON file format and copy it to a local Git repository, from which you can perform Git commands to push the files.

First Solution

First, we need to create a Grafana service account token. Go to Navigation → Administration → Service accounts → Add service account:

You need to download the Python extension for Visual Studio Code. The link to it is mentioned in the references section.

NOTE: You can refer to the code in the GitHub repository.

First, create a GitHub access token. Go to your account’s profile page and select Settings → Developer Settings → Personal access tokens → Tokens (classic) → Generate new token:

Select the scope you want for the token and generate the token:

repo: Full control of private repositories
repo:status: Access commit status
repo_deployment: Access deployment status
public_repo: Access public repositories
repo:invite: Access repository invitations
security_events: Read and write security events

Make sure to copy the token.

Subsequently, fill out the required information in the script : access_token,repository_ownerand repository_name.

Now, create a Python file and copy the following script that will download the Grafana dashboard in JSON format to the mentioned folder path

Run the script and, once completed, check the GitHub repository to see if the files were properly uploaded. Here’s the explanation of this script:

It imports the required libraries.
Defines the Grafana API URL, service account details, and the directory where exported dashboards will be saved.
Sends a call to the Grafana API for a list of dashboards.
Determines whether the directory for saving JSON files exists, and if not, creates it.
Iterates through each dashboard, retrieving its JSON data and saving it locally under a name.
It then connects to GitHub using an access token and gives the repository and folder path.
It lists all files in the provided directory, excludes JSON files, and generates an array of local file paths.
It reads each JSON file’s content, encodes it in base64, and prepares it for submission to GitHub.
It sends a PUT request to the GitHub API to create or update a file in the repository’s specified folder.
If the file is successfully created, it prints a success message; otherwise, it prints “Already Exists”.

Second Solution

Copy the script below to a file in Visual Studio Code. Fill out the required information in the script : access_token,repository_ownerand repository_name.

After that, install the Git extension in Visual Studio Code and configure the GitHub login. Then, configure the repository username:

git config –global user.name “YourUserNameOnGitHub”

And the user email:

git config –global user.email “YourEmail”

Now, go to the desired path, initialize the Git repository, and clone it:

git init
git clone “GitHubRepoUrl.git”

There are various ways to copy files in Python, some of them are mentioned below:

shutil.copy(): Copies a single file.
shutil.copyfile(): Copies the file contents.
shutil.copytree(): Copies the entire folder.

Theshutil module in Python allows us to perform high-level file operations.

Run the script and, once completed, ensure that all files have been copied to the specified folder. As a last step, add the new files to the GitHub repository:

git add . (to add all the files)
git commit -m “commit message” (to commit all the changes)
git push (push the committed files to the git repo)

Here’s the explanation of this script:

It imports the required libraries.
Specifies the Grafana API URL, service account credentials, directory to save exported dashboards, and destination directory for backups.
Sends a call to the Grafana API for a list of dashboards.
Determines whether the directory for saving JSON files exists, and if not, creates it.
Iterates through each dashboard, retrieving its JSON data and saving it locally under a name.
It then creates a timestamp using ‘datetime.datetime.today()’ to uniquely identify the backup folder.
Copies the directory containing exported JSON files (‘graf/’) to the destination directory (‘status-dashboard/grafana_bkp:’) and add the timestamp to the folder name.
Finally, it runs Git commands (add, commit, and push) from the terminal to upload the backed-up files to a Git repository.

Conclusion

Grafana is a popular monitoring choice that supports multiple data sources, various visualization options, alerting, management, and other features. And most importantly, Grafana does not store any data. With this script, you will be able to store the data in Grafana and upload it to your repository as well.