Data Streaming with webKnossos Libs

A while ago we released our open-source Python library for working with webKnossos datasets and annotations. Our library makes it easy to download annotations, upload datasets to webKnossos, automate workflows and work with the various file formats produced by webKnossos.

To install it, simply run:

pip3 install webKnossos

Before the new streaming feature, you always had to either download the whole dataset from your webKnossos instance or have a local copy of the data already. Especially for large datasets, it is unfeasible to download gigabytes or sometimes terabytes of (volumetric) microscopy data.

We are happy to announce that with the latest updates you can access and interact with remote webKnossos datasets in a streaming manner. The Python webknossos library and server will do all the heavy lifting in the background using Zarr.

An example SBEM dataset of mouse cortex in webKnossos. Instead of downloading the full ~360GB you can now stream the data as needed. Explore on webKnossos.

How can I use the new data streaming feature?

To open a webKnossos dataset for streaming from its remote server use the following API call:

import webknossos as wk
remote_dataset = wk.Dataset.open_remote(“https://webknossos.org/datasets/scalable_minds/l4dense_motta_et_al_demo")
# Stream a 50x50x50 bounding box as Numpy Array
color_layer = remote_dataset.get_layer("color")
mag = color_layer.get_finest_mag()
ndarray = mag.read(relative_bounding_box=wk.BoundingBox([100, 100, 100], [50, 50, 50]))
...
# For a full download use this instead:
dataset = wk.Dataset.download(<url>)

The attentive reader will note that this interface is almost identical to the API calls for opening a dataset from your local file call (Dataset.open(…). We specifically designed the APIs to behave as similar to one another as possible.
For more information, please refer to the examples in the documentation.

We hope this feature makes automating webKnossos workflows in Python even easier and more convenient for you. If you want to provide feedback or run into trouble please let us know through a GitHub issue.