Apache Superset — Manage Custom Viz Plugins in Production

A few months ago, Superset introduced a way for anyone to create a custom viz plugin.

A viz plugin is basically a react component that integrates with a Superset dashboard, can run data queries, and display it in any visual way desired.

prerequisite for this post — how to create a custom viz plugin guide

We got excited and immediately developed our first custom plugin, and it worked! Amazing!

But it wasn’t all roses. We found out that there is no easy way to load them into a production instance of Superset and no standard way to manage custom plugins externally to superset-ui.

We started exploring our options for loading plugins:

1. Dynamic import built-in feature — at the time, this feature wasn’t mature, although today it is merged but still under a feature flag.
2. Injecting plugins into superset and rebuild docker image with plugins included inside

To be able to use them right away, we chose option 2.

The requirements were:

• Manage multiple custom plugins with the ability to reuse common code
• Auto manage plugins package versions and deployments
• Have a docker artifact based on the official version bundled with the plugins

We decided to use a “monorepo” strategy based on Lerna, yarn workspaces, and some additional Superset frontend stack that will enable us as the first step to have separated Npm packages and a good versioning mechanism for each one of our plugins.

You can read about our stack in the GitHub repository:

nielsen-oss/superset-viz-plugins

Our repository can be copied and be used as a template for anyone to start creating their custom plugins.

If you are unfamiliar with the template repository concept read about it here

Additionally, we created a CI/CD solution that not only publishes the packages but also injects the plugins into a specific version of Superset (0.38) and rebuilds a docker image.

Here is how we inject plugins:

1. Inject the dependencies into superset-frontend package.json

cd $GITHUB_WORKSPACE/incubator-superset/superset-frontend
node $GITHUB_WORKSPACE/superset-viz-plugins/scripts/addDependencies.js

2. Generate the code and inject a custom preset that contains plugins registration based

# generate preset file and locate in incubator source code
node $GITHUB_WORKSPACE/superset-viz-plugins/scripts/generatePreset.js
mv "./${PRESET_NAME}Preset.ts" "./src/visualizations/presets/${PRESET_NAME}Preset.js"
# generate setupPluginsExtra override in incubator source code
node $GITHUB_WORKSPACE/superset-viz-plugins/scripts/generateSetupPluginsExtra.js
cp "./${PLUGINS_EXTRA_FILENAME}" "src/setup/${PLUGINS_EXTRA_FILENAME}"

3. Re-run the docker build

docker build \
  -t "${REPOSITORY_NAME}:${SUPERSET_VERSION}" \
  --label "built_at=$(date)" \
  --label "build_actor=${GITHUB_ACTOR}" \
  .
docker logout
docker login --username "${DOCKERHUB_USER}" --password "${DOCKERHUB_TOKEN}"
docker push "${REPOSITORY_NAME}"

The superset docker image bundled with all of the plugins can be found here

Hopefully, You will benefit from our plugins, especially this Composed Chart — It is an XY chart that supports different visualization for different metrics

We happily accept new pull requests for improvements and new plugins as well, so feel free to contribute and improve this solution.