What’s new in Elyra 2.0
Earlier this month JupyterLab 3.0 was released, providing a slew of new features. In this blog post I’ll recap a few highlights and how the Elyra 2.0 extensions take advantage of them.
March 2021 update: Elyra version 2.1 was released, adding support for running pipelines on Apache Airflow.
April 2021 update. Elyra 2.2 adds support for R scripts, a new pipeline CLI, and is available as a Kubeflow notebook server.
August 2021 update: Elyra introduced experimental support for custom components. You can now add custom Kubeflow Pipelines components or Apache Airflow operators to pipelines.
New to Elyra? Take a look at the overview documentation. TL;DR: Elyra provides productivity features for common data science and AI tasks.
Support for Language Servers
Many modern integrated development environments simplify development by providing editor features like autocomplete, “go to definition”, “find references”, or linting.
JupyterLab optionally supports these features by integrating with Language Servers using the jupyterlab-lsp extension. Language Servers are available for many languages, such as Python and R.
Some Language Servers might not support every convenience feature.
Elyra 2.0 includes the Python Language Server, enabling these features in the notebook editor and Python script editor, as shown below.
You can add support for other languages by installing the appropriate Language Server and its prerequisites.
Visual debugger
JupyterLab includes a visual debugger front-end that supports common debugging tasks. For example, you can set breakpoints, step through code, or inspect data. The front-end requires a kernel with support for debugging.
As of January 2021 the only Jupyter Python kernel that supports debugging is xeus-python.
Do note that this kernel currently does not support all iPython features, such as cell magics.
Neither JupyterLab 3.0 nor Elyra 2.0 install xeus-python by default. You therefore have to install it separately from PyPI, conda, or build from source code.
Debugging a notebook
To debug a notebook, associate it with a debugger enabled kernel and enable the debugger. The official debugger example Python notebook provides a short introduction and is highly recommended!
Debugging a Python script
Elyra’s Python editor is also enabled for debugging.
To debug a Python script:
- Open the script in the Python editor.
- Right click on the canvas and select
Create Console for Editor. - Select a kernel that supports debugging.
- Enable the debugger in the console.
- Set one or more breakpoints in the source code.
- Run code to breakpoint and step through the code.
Elyra 2.0 also includes several enhancements that are not specific to JupyterLab 3.0.
Pipeline editor enhancements for Kubeflow Pipelines
The visual pipeline editor, which I’ve introduced in this blog post for Kubeflow Pipelines and this blog post for local execution in JupyterLab, has been extended to support custom resource allocation and the Tekton workflow engine.
Customize resource allocation
When a pipeline node is executed on Kubeflow Pipelines, it is processed in a container. The amount of CPU, GPU, and memory (RAM) resources that are allocated for the container is governed by configuration settings in Kubernetes.
With the latest enhancement in Elyra you can request additional CPU, GPU, or memory resources to speed up processing of your notebook or Python script during pipeline execution. You configure those resource requirements in the node’s properties view, as seen in the example below.
If you leave the properties empty, the Kubernetes default settings are applied.
Support for Tekton workflow engine
Kubeflow Pipelines utilizes a container-native workflow engine to execute pipelines. As of January 2021 Argo and Tekton are the only two supported engines.
When you run (or export) a pipeline in the Elyra pipeline editor, it is compiled into a workflow engine-specific format. In previous releases Elyra could only produce Argo-compatible outputs, but this restriction is lifted in version 2.0. Since Elyra cannot automatically determine which workflow engine your Kubeflow Pipelines installation leverages, you need to specify the type when you create a runtime configuration for your Kubeflow Pipelines deployment.
If you don’t specify an engine type in your configuration, Argo is assumed as default to retain backward compatibility with older Elyra releases.
If you are unsure what engine your Kubeflow Pipelines deployment utilizes, check with the administrator. If you select the wrong engine type your pipelines will fail to execute.
Code snippets
Elyra 2.0 makes it easier to create code snippets. The code snippet editor now allows for the creation of code snippets from highlighted code.
To create a code snippet within an editor select the desired piece of code, right click, and choose from the context menu Save as Code Snippet.
To learn more about other enhancements or bug fixes that went into Elyra 2.0 please check the changelog.
How to get started
If you are new to Elyra, it’s easy to get started.
Install from PyPI or conda
The Elyra JupyterLab extensions can be installed from PyPI, conda, or from source code. Refer to the installation guide for details.
Quick side note. JupyterLab 3.0 introduces a new extension type named prebuilt extensions. These extensions don’t need a JupyterLab rebuild or a Node.js installation. Unfortunately we are unable to distribute Elyra 2.0 as prebuilt extensions, but are considering doing it for future releases.
Run JupyterLab/Elyra in a container image
The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.
Getting involved
During the past year we have received many suggestions and feedback from users like you. We appreciate your input and encourage you to stay involved. Open GitHub issues, reach out on gitter, post in our new forum, or join our weekly community meeting.
Use the forum to ask general questions or share things you’ve created, like Nick Bourdakos did. He recently published an awesome pipeline viewer for Google Chrome and Mozilla Firefox that visualizes pipeline files on GitHub, like shown in the screen shot below.
The Elyra community has lot planned for the first half of this year. Subscribe to our medium publication or follow us on Twitter to stay in the loop.
