A Private Meltano Hub for Enterprise Users

What is the Meltano Hub?

Meltano Hub is an index of all Meltano plugin definitions. Meltano references the hub for certain operations like adding, updating and installing plugins.

Why Meltano Hub?

When using Meltano for the first time, all 600+ (at the time of writing) connectors and tools are immediately available for a user to run with just a few commands.

# initialise a new project
meltano init my-project
cd my-project

# add plugins
meltano add extractor tap-spotify
meltano add loader target-jsonl

# set required configuration
echo TAP_SPOTIFY_CLIENT_ID='<your client id>' >> .env
echo TAP_SPOTIFY_CLIENT_SECRET='<your client secret>' >> .env
echo TAP_SPOTIFY_REFRESH_TOKEN='<your refresh token>' >> .env

# run sync
meltano run tap-spotify target-jsonl

This makes Meltano highly accessible for new users with the goal of moving data from some source to some destination. Additionally, all major databases are supported out-the-box.

Why not Meltano Hub?

Problem

Since Meltano Hub is hosted over the internet and has community-supported plugins, if you are working in an enterprise setting where network restrictions and/or other security policies may be in place, you may run into issues when adding, updating or installing plugins in your project.

Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Could not connect to Meltano Hub. Could not reach Meltano Hub.

Some other considerations: does your security policy allow you to update code from an unknown source? What about updating your own set of internally supported plugins? What about authentication to the hub?

Solution

Fortunately, Meltano does provide a means to override the Meltano Hub connection, assuming that you have access to an alternate Meltano Hub — that’s where we think Matatika can help!

Matatika Community Edition as a Private Meltano Hub

The Matatika CE can function as a fully working Meltano Hub from your local machine. The CE comes with a wide range of plugins available out-the-box (500+ at the time of writing) sourced from Meltano Hub— we’ve removed a couple we don’t yet support and there are a number of plugins that are unique to Matatika.

We are actively working on ways to provide tested and supported versions to the whole community where plugins don’t have enterprise-grade community support — get in touch if this interest you!

Adding custom plugins and getting started

Follow the steps here to get started. Once the CE is running, set the following configuration for Meltano:

# all supported plugins
echo MELTANO_HUB_API_ROOT='http://localhost:8080/api' >> .env

# all installed plugins for a workspace
# echo MELTANO_HUB_API_ROOT='http://localhost:8080/api/workspaces/<your workspace id>' >> .env

# profile menu > "API Keys" > "Developer Token" 
echo MELTANO_HUB_URL_AUTH='Bearer <your matatika api token>' >> .env

All done! You can now add, update and install plugins from your own private Meltano Hub using the Matatika CE.

To add or update your own plugin definitions, simply copy a plugin .yml file into the appropriate plugin type directory in the matatika-ce/plugins directory. This will get deployed and become available to meltano add to your project immediately.

Found out more at matatika.com