Mimicking MongoDB Plugin in Grafana for Free: A Workaround

Grafana is an open-source analytics and interactive visualization web application. It allows users to ingest data from a wide range of data sources, query this data and display it using beautiful customizable charts for easy analysis, helping you draw better conclusions out of it to make decisions around that data.

The Prelude

Now I had data stored in a conventional document object model in my Mongo Database and on a quick google search I figured out hey, we can actually write mongo queries in Grafana to visualize my data.

Or so I thought.

I stumbled upon the MongoDB Datasource plugin for Grafana while browsing resources. Was overjoyed and finished my POC on a temporary Grafana Cloud instance. Things worked as expected. I could easily write aggregation queries into Grafana and have time-series and bar charts being plotted out for me on my dashboard. What I really missed out on was a little label on the documentation of this plugin.

It said Enterprise. And since Grafana cloud comes with most Enterprise plugins as a part of the offering, I never realized it would be a problem when we shifted to our self-hosted OSS version of Grafana. I happily setup things only to be welcomed by this notorious error message “Error starting the plugin” which, and note, I thought I COULD FIX IF I PERSISTED long enough. Rookie mistake.

Thanks to the senior who pointed this out

Well, storytime over, we HAD to find a workaround. Some possible solutions were to either move our solution to a database whose plugin was supported by Grafana OSS or, something GPT-4 suggested us to do.

Finding a Workaround

AI told us, hey since you already have a server with access to your Mongo database, why don’t you use one of the free JSON plugins Grafana provides to send JSON responses from APIs that this plugin can then render.

Um okay. I kinda skipped sleeping that night and built those custom query endpoints out. Adding the code here incase someone ever comes across the same issue.

Kudos to the creator of the Infinity JSON plugin for Grafana.

1. Pretty simple to install following the documentation. If required modify the path to plugins in your grafana.ini or defaults.ini file and Grafana should be able to read the plugin given adequate permissions. Here I added a simple health check ping route from my local Sanic (python) server.

2. Create a new dashboard, select Infinity as your datasource and then create a new panel. Add your API endpoint that would serve JSON data. You could choose either GET or POST requests depending on your requirement.

3. Now I wanted the panel to look very similar to what I was doing with the Mongo Datasource Plugin, I used POST and passed my aggregation pipeline as I would for MongoDB and passed it in the JSON body to my API. Also added a few custom keys like collection and range to make handling things on my server easier.

4. On the server end you gotta define 2 endpoints that grafana uses, which I defined as follows. You could add their extended functionalities but my specific usecase didn’t require me to.

@grafana.post('/search')
async def search(request) -> HTTPResponse:
    collections = await mongo.db.list_collection_names()
    return sanic_json(collections)

@grafana.get('/annotations')
async def annotations(request) -> HTTPResponse:
    return sanic_json([])

And then define the API endpoints that your Grafana panel would hit:

@grafana.post('/query_custom')
async def query_custom(request: Request) -> HTTPResponse:
    data = request.json
    collection_name = data['collection']
    pipeline = data['pipeline']
    _from = data.get('range', {}).get('from')
    _to = data.get('range', {}).get('to')

    match_filter = {}
    collection = db.get_collection(collection_name)

    _team = await cls.get_team() # some misc. function

    if _team and _team!='all':
        # some custom logic to update our match filter
        _members = await UserManager.get_members(_team)
        if collection_name == 'members':
            match_filter = {
                'author.display_name': {
                    "$in": _members
                }
            }

    if _from and _to:
        _from = datetime.fromtimestamp(int(data['range']['from'])/1000)
        _to = datetime.fromtimestamp(int(data['range']['to'])/1000)
        if collection_name == 'members':
            match_filter["created_at"] = {"$gte": _from, "$lt": _to}
        elif collection_name == 'builds':
            match_filter["start_time"] = {"$gte": _from, "$lt": _to}
        else:
            match_filter["date"] = {"$gte": _from, "$lt": _to}

    existing_match = pipeline[0].get('$match', {})
    match_filter.update(existing_match)
    if existing_match:
        pipeline[0]['$match'] = match_filter
    else:
        pipeline.insert(0, {"$match": match_filter})
    
    results = collection.aggregate(pipeline)

    class DateTimeEncoder(json.JSONEncoder):
        def default(self, obj):
            if isinstance(obj, datetime):
                return obj.isoformat(' ')
            return super(DateTimeEncoder, self).default(obj)

    response = [doc async for doc in results]
    return sanic_json(response, dumps=lambda obj: json.dumps(obj, cls=DateTimeEncoder))

You could use a different way to handle your DateTime objects. And then maybe add custom key-value pairs in the request body to suit your logic better. Another way could be to define different endpoints for different queries altogether:

@grafana.get('/get_users')
async def get_users(request: Request) -> HTTPResponse:
    results = await User.distinct("display_name")
    return sanic_json(results)

This may or may not take params and simply return you JSON responses for the different types of visualizations you would want. You can also use the same API to be able to create variables on your dashboard.

Here the user variable would be rendered as a dropdown selector with all distinct user’s names.

Benefits of This Approach

1. Cost-Effective: Utilizes free plugins and avoids the need for the Enterprise version of Grafana, reducing costs significantly.
2. Flexibility: Allows for highly customizable queries tailored to specific requirements. Easily integrates with existing systems and can adapt to various data sources.
3. Scalability: Can be extended to handle more complex queries or additional data sources without significant changes and these API endpoints can be reused and extended for different visualization needs.
4. Independence and Enhanced Security: Independence from cloud services ensures better control over data and infrastructure. Self-hosting also ensures data privacy and security controls are entirely under your administration. Your API endpoints can be secured using authentication and authorization mechanisms as needed.

Conclusion

Implementing a custom solution using the Infinity JSON plugin for Grafana is a clever and practical workaround for visualizing MongoDB data without requiring the Enterprise version of Grafana. This approach offers significant benefits by leveraging API endpoints to fetch and manipulate data.

Happy visualizing!