QuantStack: 2021 in review
Let’s look back at what we have accomplished in 2021!
This was a crazy year of innovation for the open-source team at QuantStack. From JupyterLab to the packaging ecosystem and high-performance computing, we made some major strides in all areas, while almost doubling the size of the team.
Collaborative Editing in JupyterLab
One of the major developments of the team in 2021 was the support for collaborative editing in JupyterLab, which was released with JupyterLab 3.1.
For this project, we partnered with Kevin Jahns, the main author of the Yjs library, a fast and scalable CRDT implementation providing a complete toolkit for creating collaborative applications. The JupyterLab RTC team at QuantStack (Carlos Herrero, Jeremy Tuloup, David Brochart, and Duc Trung Le) is now making progress towards a more integrated multi-user experience in JupyterLab, and more importantly towards enabling collaborative editing workflows for other types of applications, such as CAD modellers.
The original announcement for the support on collaborative editing is available here.
The JupyterLab Visual Debugger
One of the main additions to JupyterLab by the QuantStack team was the visual debugger. Initially built as an extension, the visual debugger has become a core component to JupyterLab 3.0, which was released in early 2021.
The first Jupyter kernel to support interactive debugging was Xeus-python, a Python kernel, but QuantStack brought debugging support to the ipykernel and xeus-robot kernels.
Beyond adding debugger support to new kernels, the QuantStack debugger team (Johan Mabille, Jeremy Tuloup, Sylvain Corlay, Martin Renou) has added many new features to the JupyterLab debugger, including the ability to execute code when stopped at a break point. We are now working towards supporting multi-threaded debugging.
RetroLab and the future of the Jupyter notebook
JupyterLab is the next-generation web-based user interface for Project Jupyter, and provided from the start the ability to display multiple panels on the page (such as code editors, notebooks, file explorers, terminals, and more) as it was a commonly requested features. However, an equally large number of users were attached to the full-page notebook UI provided by the classic notebook.
This is the raison d'être of RetroLab, a JupyterLab distribution with a "retro" look and feel, more similar to the classic Jupyter notebook. Built with JupyterLab components, RetroLab takes advantage of all of the benefits of using the JupyterLab architecture with respect to collaborative editing, visual regression testing, accessibility, internationalisation, debugger support, and the broad ecosystem of JupyterLab extensions (mime renderers, widgets, language server protocol).
Since late November, RetroLab became more central to the Jupyter roadmap, with the Jupyter Enhancement Proposal by Fernando Perez, to base the next version of the Jupyter notebook upon RetroLab.
RetroLab was created by Jeremy Tuloup, who has now been joined by community members working on the plan to build the next major version of the notebook with the RetroLab codebase.
JupyterLite
JupyterLite is a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions.
In-browser Jupyter kernels, which can be installed as plugins, now include a Python kernel called "Pyolite" that leverages the Pyodide in-browser scientific Python distribution, and the IPython shell.
Pyolite offers a comprehensive support of ipykernel features (magics, interactive widgets, rich display, Matplotlib support). In addition to the Pyolite kernel, we are now working on new JupyterLite kernels based on Xeus, including a Lua, xeus-lua.
JupyterLite was started by Jeremy Tuloup, and now receives significant contributions from community contributors such as Nicholas Bollweg (who created a similar project in the past called Jyve), but also other QuantStack developers such as Madhur Tandon and Martin Renou. Thorsten Beier is also making strides towards packaging more Xeus-based kernels in WebAssembly.
JupyterLite also provides in a "classic" flavour, based on the RetroLab distribution. Collaborative editing is made possible thanks to the WebRTC YJS provider.
We have big plans for the future of JupyterLite and in-browser computing, stay tuned for future developments!
nbterm
nbterm is a light-weight implementation of a Jupyter notebook UI in the terminal. It fills a hole in the terminal-based Jupyter applications offering, as we only had a console (e.g. IPython). Now you can open, edit and execute notebooks right from your terminal! See this blog post by David Brochart for the announcement.
Jupyter kernels
Year 2021 was the year of Xeus-based kernels! There are new kernels to the family:
- Xeus-robot, a Jupyter kernel for Robot Framework, enabling rich robotic process automation in Jupyter, and supporting the interactive debugger.
Xeus-robot was co-authored by Martin Renou and Johan Mabille. - Xeus-SQL, a Jupyter kernel a general SQL implementations based on the native implementation of the Jupyter protocol Xeus and SOCI, a database access library for C++. Xeus-SQL was created by Mariana Meireles and makes use of xvega (by Madhur Tandon) for inline data visualization of query results base on VegaLite.
- Xeus-Lua, is a Jupyter kernel for Lua, supporting rich display and interactive widgets.
A major milestone for the Xeus stack was the release of Xeus 2.0, which enables a cleaner split between the language-specific component, and the protocol-specific component of the library, allowing for overriding the transport layer of the kernel protocol. This enabled the use of Xeus for creating pure in-browser kernels for JupyterLite.
In the near future, we will make it easier to embed Jupyter kernels in desktop applications, with a new approach for event loop integration.
akernel
akernel is a Python Jupyter kernel that explores novel execution models:
- asynchronous execution: cells can run concurrently using
async/await. That opens new possibilities in dashboarding using e.g. Voilà, where different parts of the dashboard can be executed concurrently. - reactive programming: notebook cells update any time a cell’s value changes, enabling out-of-order execution a la Observable notebooks.
Jupyter widgets
QuantStack is very active in the Jupyter widgets ecosystem, and maintains some of the most popular Jupyter widget libraries, such as bqplot, ipyleaflet, ipygany, ipycanvas, and many more.
The entire Jupyter widgets ecosystem has made significant strides in 2021, as we are nearing a major release of ipywidgets (version 8.0) with many significant improvements.
Pre-releases of ipywidgets 8.0 are already available for download if you wish to try them.
More broadly, we have generalised the visual regression testing of the Jupyter widget libraries, making the entire ecosystem more robust, thanks to the Galata UI testing framework that is now integrated into JupyterLab.
Ipycytoscape was transfered to Cytoscape GitHub organization
The ipycytoscape package, which was started by Mariana Meireles in 2020 (and who is still leading the development) has been transferred to the Cytoscape GitHub organization, which is governed by the Cytoscape consortium.
This is one more example of QuantStack's commitment to multi-stakeholder open-source software development.
Ipympl
IPympl is a Jupyter widget enabling the interactive use of Matplotlib figures in Jupyter frontends (such as JupyterLab, Voilà, RetroLab, or the classic notebook). It is the successor of the "nbagg" Matplotlib backend.
In 2021, we furthered the development of ipympl significantly, enabling the creation of a static version of the figures as images in the notebooks, allowing its rendering in e.g. GitHub.
Jupyter Games!
Making their own tiny video game can be a great way for kids to learn programming in a playful matter. While Jupyter is widely used as an educational tool, enabling simple 2-D game development and let kids develop games like a billiard or a flipper in a few lines of code may be a great way to enable the next generation of developers.
This is the raison d'être of the pyb2d project, a Python wrapper for the Box2D library.
Check out the recent blog post by Thorsten Beier on pyb2b and the development of games in Jupyter.
We hope to see a lot more examples of Jupyter games, mostly for the education use case.
Towards collaborative editing with Jupyter widgets
QuantStack is now making prototypes and proofs of concepts of what collaborative editing should look like with Jupyter interactive widgets. This poses deep questions about the next-generation Jupyter interactive widgets, and how we can move from the current MVC pattern to a CRDT-based approach.
Dashboarding
Voilà turns Jupyter notebooks into standalone web applications. Built from the start from Jupyter components (kernels, the notebook format, nbconvert, JupyterLab components), Voilà is really a "Jupyter story" and has been integrated as an official Jupyter subproject.
A major improvement to Voilà made in 2021 is the ability to pre-heat Jupyter kernels and pre-render dashboards ahead of the user connections, significantly reducing the launch time of Jupyter kernel. Voilà 0.3.0 was the first stable release supporting pre-released kernel. It also incorporates the Voilà-reveal template, allowing the display of notebooks as slideshows.
Voilà-gridstack WYSIWYG editor
Voilà's templating system allows users to change the layout of their Voilà-based templates. The Voilà-gridstack template now includes a JupyterLab WYSIWYG editor for modifying the layout of dashboards.
The WYSIGYG editor was developed by Carlos Herrero and Jeremy Tuloup.
The development of the Voilà-gridstack editor triggered important changes in the upcoming ipywidgets 8.0 release with respect to sharing widget managers between notebook views.
Jupyverse
Jupyverse is an opinionated reboot of the Jupyter Server project. Built as a collection of plugins for FPS. The motivation for this reboot are multiple:
- The current jupyter server is based on Tornado, which is not very actively maintained anymore. We based FPS and Jupyverse on the very active FastAPI project, which also offers significant improvements in terms of performance, but also automatically generated OpenAPI documentation.
- We were able to use the same plugin-based architecture as JupyterLab, making even core components of the servers in the form of plugins. This approach will allow us to produce the same kind of “remixes” of extensions for putting together new applications.
Jupyverse is built from the ground up with large-scale deployments and collaborative editing in mind, with the finer-grained permissioning model that this will require.
FPS was created by Adrien Delsalle, and Jupyverse by David Brochart.
Improving JupyterLab's reactivity
Notebooks of hundreds to a couple of thousands cells are more and more common nowadays and they can contain as many interactive plots. Unfortunately those very large notebooks make the UI very slow and frustrating for the users. In the later months, Frederic Collonval improved benchmarking large notebooks handling by JupyterLab by adding a benchmark CI job to track PR’s impact and another weekly more in-depth test in the side project jupyterlab/benchmarks.
In parallel to the monitoring improvement, various ways to improve the UI reactivity were tested. A simple enhancement to fasten tab switching was merged in JupyterLab. And the most promising path from the proof of concepts tested is the notebook windowing approach. It requires some important rework so stay tuned for more on that subject next year.
Beyond Jupyter, QuantStack is very active in the low-level tooling for scientific computing. One of the main components maintained by QuantStack is Xtensor, a C++ N-D array library with an API very similar to that of NumPy, with broadcasting and lazy computing.
Xsimd
Xsimd is a project that was split from Xtensor and which provides a unified C++ API for performing SIMD operations, and supporting a large number of SIMD instruction sets:
- For x86 architectures: SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, FMA3, AVX2, AVX512, SSE4A, FMA4, XOP.
- For ARM architectures: ARMv7, ARMv8.
Since it was split from Xtensor as an independent project, Xsimd has grown considerably in popularity, and major open-source projects have adopted the package such as Apache Arrow and Pythran.
In 2021, we performed a major refactoring of XSimd, and released XSimd 8, which allows for the creation of "fat binaries" that can handle multiple instruction sets, and allows for the selection at run time. The XSimd 8 refactor was led by Johan Mabille and Serge Guelton.
QuantStack is seeking funding and help to enable new architectures on XSimd, such as POWER architectures.
Xtensor-Zarr
Xtensor-zarr is the latest family member of the Xtensor stack. It provides a C++ reference implementation of the Zarr core protocol (versions 2 and 3) based on xtensor and zarray. The development of Xtensor-zarr was conducted by David Brochart and Johan Mabille.
Mamba everywhere!
Mamba is a reimplementation of the conda package manager in C++, offering significantly better performances than the reference implementation. A lot of attention is also going into providing better error messages for the users and improving upon conda's stability.
In 2021, we have seen the adoption of mamba grow considerably, with conda-forge, Read the Docs, and MyBinder adopting mamba for its improved performances and resource usage. Mamba is now a central component of the conda ecosystem.
Last, but not least, another distribution of mamba has the form of a standalone single-binary executable called "micromamba" that only weighs 5MB and can be used to create conda environments, without requiring a full miniforge / miniconda download that weighs as much as 65MB.
Quetz
Quetz is an open source server for conda packages. It is built upon FastAPI with an API-first approach. Quetz has a fine-grained permission model on channels and packages, enabling the type of separation of concerns required for large channels with many stake holders like conda-forge.
In 2021, we furthered the development of the Quetz server, with a new extensible frontend built from the same extension system as JupyterLab, making Quetz a very pluggable with both front-end and backend plugins.
A collection of plugins is available, allowing use cases such as:
- the transmutation of packages upon upload,
- solving environment as a service,
- converting PyPI wheels to conda packages on the fly,
- GitHub-style package and channel search,
- metrics dashboards with prometheus and Grafana.
More importantly, Quetz made the notion of mirror a central feature, allowing organizations to host a local Quetz instance mirroring an external channel, alongside a local private channel.
Package signing in mamba
One of the main innovation in the mamba package manager in 2021 is the implementation of TUF (The Update Framework) protocol in mamba, in order to enable package signing and verification. Supply chain security will be key to the broader adoption of the conda ecosystem, and especially of conda-forge and will the rise of networks of mirrors.
Check out the blog post on supply chain security by Adrien Delsalle, who implemented the TUF protocol in Mamba.
Rhumba
A major benefit of having implemented mamba in C++ is that it can be easily embedded into other programming language without requiring a Python interpreter. This is what we have done with the Rhumba package, a fast R package manager powered by mamba.
The use of the thousands of readily-available R packages from conda-forge package with Rhumba makes a very decent R distribution, with much better performances than CRAN.
For a simple comparison, installing the devtools package with Rhumba on my laptop takes under 30 seconds, while doing it with CRAN on the same conditions takes 12 minutes. Rhumba was primarily developed by Mariana Meireles with significant contributions by Adrien Delsalle, who made a pure C API for mamba for this purpose. Check out Mariana's Blog post post about Rhumba for more details.
PackagingCon
One of the highlights of 2021 was the first instalment of PackagingCon. PackagingCon is a conference for developers of software package management software, as well as software packagers and communities.
Led by Wolf Vollprecht, the organizing team was able to gather more than 300 participants, for a series of 67 talks. Many packaging ecosystems were represented, including conda / mamba, DNF, Nix. The event, which was supported by NumFOCUS, received funding from Citadel, Bloomberg, CloudSmith, QuantStack, OpenSUSE, Quansight, Anaconda, Tweag, Collabora. Not bad for a first event!
CZI grant for conda-forge
As announced by the Chan Zuckerberg Initiative the mamba project has been selected for funding as part of the essential open source software for science, grant cycle 4 (EOSS-4)! Work on this grant will begin in January 2022, with the following goals:
- A faster package management experience
- Improved error message handling
- Enabling package signing for conda-forge
- Increase the sandbox-ability of virtual environments
- Organizing PackagingCon 2021 and 2022
For more details, check out our blog post announcing the news.
The RoboStack ROS distribution
The Robot Operating System (ROS) is an open-source collection software libraries and tools that help build robot applications. As ROS has become a de facto industry standard, getting a functional ROS development environment can be a challenge as the official distributions of ROS are bound to specific Linux distributions and versions.
RoboStack is a cross-platform conda-based distribution of ROS, which can be used to install ROS on any Linux distribution, as well as on Windows and OSX. As RoboStack is growing in adoption and popularity, it has become a major motivation for some of the latest innovation in mamba, especially with respect to building a large number of inter-dependent packages in one go.
JupyterLab-ROS
JupyterLab-ROS is a collection of JupyterLab extensions for better integration with the ROS ecosystem. Built on top of RoboStack, JupyterLab-ROS brings together the best of the ROS and Jupyter ecosystem, to provide an advanced robotics cloud command station.
Jupyter distinguished contributors award
In 2021, three QuantStack team members were honored by the Jupyter Distinguished Contributor (JDC) award: Johan Mabille, Martin Renou, and Jeremy Tuloup.
QuantStack leadership
The QuantStack team has grown significantly in the past two years, and from a small group of open-source developers to a significantly larger team.
Leadership is taking shape as Wolf Vollprecht became the Chief Technology Officer of the company, while Jeremy Tuloup, Johan Mabille, and Martin Renou are now technical directors. Finally Fanny Loustau-Chartez joined the leadership team as Chief Financial Officer.
We are recruiting!
We have several open positions listed on our website. Join an amazing team of open-source developers and help us build the common infrastructure of scientific computing.
QuantStack allows for full remote work, but also has offices in Berlin and in the Paris area! We are looking for candidates who can work in France or Germany.
