Enabling the Elyra pipeline editor for specific runtimes

Sunset on Usedom, an island in the Baltic Sea

The Elyra Visual Pipeline editor supports three runtimes: local pipeline execution in JupyterLab, remote pipeline execution in Kubeflow Pipelines, and remote pipeline execution in Apache Airflow. By default support for all the runtimes is enabled when you install Elyra.

Runtime-specific support is enabled if a corresponding editor tile is displayed in the JupyterLab launcher.

Starting with Elyra version 3.15.0 you can enable support for specific runtimes and essentially hide tasks that are not related to that runtime.

This installation of Elyra is configured to support only Kubeflow Pipelines as runtime

For example, if only support for Kubeflow Pipelines is enabled, the JupyerLab GUI will only expose the user to these tasks:

• run a Jupyter notebook, Python, or R script on Kubeflow Pipelines
• manage [Kubeflow Pipelines] runtime configurations
• manage [Kubeflow Pipelines] component catalog connectors
• create, run, or export [Kubeflow Pipelines] pipelines

The corresponding tasks for other runtimes are hidden.

Enable specific runtimes to improve the user experience. In this example users can only run a Jupyter notebook in Kubeflow Pipelines. Apache Airflow is not listed as an option.

Enabling support for specific runtimes

You can enable support for specific runtimes by specifying a parameter when launching JupyterLab or customizing the Elyra configuration file.

Enabling support using command line parameter

To enable support for a specific runtime launch JupyterLab with the PipelineProcessorRegistry.runtimes parameter, specifying local, kfp, or airflowas parameter value.

For example, to only enable the Kubeflow Pipelines runtime support, run

jupyter lab --PipelineProcessorRegistry.runtimes=kfp

To enable support for multiple runtimes, say local execution in JupyterLab and Airflow, specify the parameter multiple times

jupyter lab --PipelineProcessorRegistry.runtimes=local --PipelineProcessorRegistry.runtimes=airflow

To avoid the need to specify this parameter you can also customize the Elyra configuration file.

Enabling support using the Elyra configuration file

To customize the Elyra configuration file:

1. Stop JupyterLab if it is running.
2. Generate the jupyter_elyra_config.py configuration file.

 jupyter elyra --generate-config

Note: You must specify `elyra` as the `jupyter` subcommand instead of `lab`.

3. Open the generated (Python) configuration file.

4. Locate the PipelineProcessorRegistryconfiguration section.

#------------------------------------------------------------------
# PipelineProcessorRegistry(SingletonConfigurable) configuration

5. Locate the configuration entry for PipelineProcessorRegistry.runtimes, which is commented out by default.

# c.PipelineProcessorRegistry.runtimes = []

6. Remove the leading # character and add kfp , airflow, or local entries as desired.

c.PipelineProcessorRegistry.runtimes = ['local', 'airflow']

7. Save the customized configuration file.

8. Start JupyterLab. The pipeline editor tiles for the specified runtimes are displayed in the launcher window.

Conclusion

As an open source project we rely heavily on input from the community. The feature I’ve outlined in this post is a great example of such input. Find information about how you can reach out with questions and suggestions here, or learn more about contributing here. If you are new to Elyra and want to learn more check out our resources.

Thanks for reading!