A User Interface for Meltano

Last month we announced our Community Edition, a free version of our platform for data teams to evaluate and develop with Matatika. Once we had the community edition up and running, we soon realized that a read-only version of our lab would make a perfect UI for Meltano projects.

The Matatika Plugin vs Community Edition

Immediately we saw there were slightly different requirements for a Meltano UI compared to our Community Edition. First the Community Edition is all of our features and functionality of our cloud data platform. This is great if you need a robust data solution that can create and manage multiple data projects in a single platform, (complete with API, DataOps, and all the bells and whistles), but for a Meltano developer UI it needs to be quicker and simpler.

We went in the direction of being a read only UI, this solved some user permissions issues we were running into with docker editing Meltano project files, as well as letting us cut down the number of docker images running. To help keep things simple we disabled some of the functionality around creating and editing Matatika pipelines, and added support for Meltano schedules in their place. Now if you start up the Matatika Lab in a Meltano project with schedules, you see them all visualized in the pipelines screen.

Once the Matatika Lab is running in your Meltano project, you get to continue CLI development, adding plugins or schedules and they will appear in the Lab UI. You also get some quick visualization out of the box, which we will talk about in the next section.

A GitHub Example Project

Our Matatika plugin was built alongside an example GitHub Analytics project. We used this project as our guiding light for what we would want from a read only Meltano UI and what we could do to make the experience better. And you can clone our GitHub Analytics example project from GitHub and get it running in minutes!

The project has everything set up to get running right away, with a short list of steps in the project’s readme. All you need to provide is a GitHub Classic Personal Access Token, which we also go over how to get and scope correctly when you go to fill in the setting or again in the project’s readme. Once provided, you can either choose a GitHub repository or run the pipeline right away and get some test data from Meltano’s GitHub repository.

Once the pipeline has finished running you can head to the Datasets page in the lab and expand out any of them to see a chart or table. These datasets live in the example project under analyze/datasets/tap-github, and using our Dataset Documentation, you can adjust them or create your own.

Using the Matatika Plugin

The Matatika plugin is now on the Meltano hub so it is as easy as running the following commands:

# Add the Matatika plugin to your Meltano Project
meltano add utility matatika

# Starting and stoping the lab
meltano invoke matatika lab start
meltano invoke matatika lab stop

When you start the lab for the first time some docker images will be downloaded so may take a few minutes, but once everything is started up your browser will open to localhost:3443 where you will see the Matatika Lab.

Why do this?

We use Meltano a ton, it’s a major technology in the Matatika stack, and it’s great for the technical users, who want complete control over their data project while still benefiting from a framework.

And our platform was already designed to let less technical users create a complete data project, without needing to get deep into the code.

Couple this with us seeing the occasional user in the Meltano community that just needed a simple interface to really help to get their project up and running, and we decided we could help. These users get the benefits from having a UI to get a project overview, an orchestrator built in and getting simple visualizations out of the box is the icing on the cake.

We also saw a few people working on their own projects to solve the same or a similar issue.

Specifically I want to shout out Jan Soubusta who has created a GoodData example project also for GitHub and Jacob Matson with the MDS in a box project. Both well worth checking out to see other ways to go about end to end Meltano projects.