Zarr Streaming from webKnossos

As some of you might know, we are working closely with the Zarr community on adding full Zarr API and OME-Zarr support to webKnossos. In this blog post, I will share some progress on streaming any dataset from webKnossos to compatible third-party programs through the Zarr interface.

In parallel to our work on loading Zarr datasets from remote sources (cloud storage, HTTP, local disk — more on that soon), we have expanded webKnossos such that any dataset and annotation in webKnossos can be shared, accessed and streamed to other third-party programs and libraries that understand the Zarr interface. Notable examples include streaming to Neuroglancer, Fiji and Napari.

A webKnossos dataset viewed in Neuroglancer and streamed through the new Zarr API.

We believe that this move provides the broadest amount of choice to our community in choosing the right tool for the right job and maximizing the interoperability between the various scientific tools. While you were always able to download any annotation from webKnossos to work with in other programs, this switch to the Zarr interface makes it even easier.

For those interested in the technical details, we implemented the OME-Zarr NGFF spec v0.4 with full support for dataset streaming, loading subsets of data, image pyramids/LoD, etc.

Why should I use webKnossos to host my Zarr datasets?

While there are several methods for hosting Zarr datasets, e.g. on cloud-storage such as S3 or Google Cloud, we believe that webKnossos offers some excellent benefits for hosting Zarr datasets:

• webKnossos is great for organizing datasets. All your datasets are gathered in one central hub where you can tag and search for them.
• webKnossos comes with authorization and user roles. Only datasets and annotations that are shared publicly can be accessed through the Zarr interface.
• webKnossos can securely host any selected, private volume annotation. No need to share any credentials to your server or cloud buckets for private sharing.
• webKnossos has all the tools to immediately view and more importantly annotate (large) volume datasets already built-in. Any modification to annotations/segmentations made in webKnossos will show up in third-party tools.

How can I stream my webKnossos datasets through the Zarr interface?

Any webKnossos dataset that is configured to be publicly accessible, can be accessed through the OME-Zarr API. The Zarr web links can be found in the webKnossos UI for your convenience:

1. Open any public dataset annotation from your dashboard (“View”)
2. From the dropdown menu next to the “Save” button, select “Zarr Links”
3. Click on the “Copy” button for the displayed Zarr link
4. Select a layer from the dropdown.

Zarr links to public datasets can be found and copied directly from the webKnossos user interface.

Notes:

• Due to the nature of the Zarr spec, each Zarr web link can only point to the data for a single dataset layer, e.g., a “color” layer containing the microscopy data.
• To privately share a dataset, see the instruction for annotations below.
• Example: https://data-humerus.webknossos.org/data/zarr/scalable_minds/l4dense_motta_et_al_demo_v2/color

How can I stream my webKnossos volume annotations through the Zarr interface?

webKnossos can also expose any of your volume annotation, i.e., segmentation layers, through the OME-Zarr interface. Since webKnossos annotations typically belong to a single user, we consider them to be private resources that should not be available without any form of authentication. Therefore, webKnossos exposes annotations together with an authentication token and users can set a time duration for how long this resource should be available.

You can find and configure the Zarr URLs from a new menu:

1. Open any volume annotation
2. From the dropdown menu next to the “Save” button, select “Zarr Links”
3. Create one or multiple new private Zarr links
4. Click on the “copy” icon next to the URL, select a layer from the dropdown and the Zarr link will be copied to your clipboard.

webKnossos can also share volume annotations as a private, authenticated Zarr link for use in third-party software. The GIF above shows the steps to obtain a Zarr sharing link.

Zarr links can be deleted at any time from the same interface to revoke access to the annotation. Additionally, an expiration date can be defined for each link.
Users are encouraged to create as many Zarr links as they need, i.e. one to share with a in-house colleague with a long life-time, one for sharing with an external reviewer that you potentially want to revoke at a later point in time, etc.

We hope you enjoy the ease of use and flexibility when accessing webKnossos datasets and annotations through the Zarr interface. Read more in the webKnossos docs.

One last note, we still recommend the webKnossos Python lib for downloading datasets and annotations for offline analytics/processing. Under the hood, it uses the same Zarr interface but also adds quality-of-life features and matching Python object primitives to get you started.