Google Summer of Code 2021: Development of Jupyter Kernel for MetaCall
MetaCall Jupyter Kernel is an open-source wrapper kernel that implements cross-language function calls through the MetaCall Core and the Polyglot REPL. MetaCall Core is an open-source library that brings the polyglot programming experience to Developers. With MetaCall, developers can embed different programming languages through an easy-to-use high-level API.
Leveraging IPython, this project aimed to create a kernel where the MetaCall Core can be loaded and launched through a Jupyter Notebook interface. With this Notebook, the users can try out writing and embedding code in different programming languages. It would also make it possible to use the MetaCall API and achieve the goal of bringing a unified interface for developers to try out and leverage MetaCall.
This year’s project achieved the following:
- Developed the initial minimal viable product centred around bringing a Jupyter kernel for MetaCall.
- Built and integrated a Polyglot REPL using the MetaCall Core to the IPython through a subprocess.
- Implemented and improved user-experience features centred around using the kernel with the Jupyter client.
- Implemented CI/CD pipelines for the project with GitHub actions and set-up container-based deployment with Binder.
- Created and delivered a full-fledged documentation website for the MetaCall Jupyter Kernel using Sphinx and ReadTheDocs.
Setting up the Kernel
The first step of the project was to set up the wrapper kernel using IPython and expose it on a Jupyter client. Initially decided to go with Xeus, a C++ framework used to build Jupyter kernels, we moved forward with IPython. I created a wrapper kernel, which can be built and set up using Setuptools as the build system. The initial kernel forks a MetaCall exec subprocess and loads the code on a temporary file before passing it to the exec and retrieving the standard output.
The mechanism was not sufficient to move forward with defining extra languages (we provided initial support for NodeJS), hence we moved forward with automated language inference models with
guesslang. Guesslang, provided a high-level API to automatically detect programming languages through the code. This affected the processing speed and we had to do away with the same later, in favour of user magics for faster inference.
Example usage of the MetaCall Jupyter wrapper kernel
Integrating Polyglot REPL
The second step of the project was to integrate the Polyglot REPL. For this milestone, I collaborated with @viferga and worked on a NodeJS-based REPL (Read-Evaluate-Print-Loop) which loads a Python and a Node REPL through the MetaCall’s meta-object protocol. The REPL, which makes use of a high-level API, made it possible to extend its usage as per our understanding and needs and integrate it with any client interface.
The second part of this was integrating the REPL to the MetaCall Jupyter kernel. To make this possible, we made use of a
subprocess that launches the REPL in the background and listens for standard inputs passed. This approach is commonly used by other wrapper kernels, hence we followed the same design principle. As each evaluation is passed to the kernel, the standard output from the REPL is fetched and displayed on the kernel in a clean logger output. This now allows a user to both, load, inter-change, inspect and evaluate the meta-object protocol from the kernel interface itself.
Example usage of the Polyglot REPL on the MetaCall Jupyter
Implementation of user-experience features
The third step of the project was to implement the user-experience features. These user-experience features defined the usability of the client interface and how the users can interact with the REPL. To achieve this, we decided to tweak the standard magics used on the Jupyter notebook interface, to match our requirements.
The user-experience centric features I added were:
!: Run a Shell Command on the MetaCall Jupyter Kernel
$shutdown: Shutdown the MetaCall Jupyter Kernel
$inspect: Inspects the MetaCall to check all loaded functions
$loadfile: Loads a file onto the MetaCall which can be evaluated
$newfile: Creates a new file and appends the code mentioned below
%repl <tag>: Switch from different REPL (available tags: node, py)
$loadcell <tag>: Loads a function onto the MetaCall to be evaluated
$help: Check all the commands and tags you can use while accessing the MetaCall Kernel
$available: Checks all the available REPLs on the Kernel
Implementation of CI/CD pipelines
For continuous testing, I built and managed the CI/CD pipelines for the project. The CI pipelines are built using GitHub Actions. Testing has been implemented using the
pytest module, and the CI is responsible for building and validating the source and the documentation.
GitHub Actions in action with every Pull Request and push
For documentation, I built and published the MetaCall Kernel and the source documentation on Sphinx and ReadTheDocs. Sphinx is a documentation generator for Python and was the first choice of going ahead with documenting the project. The standard documentation ensures that the community support remains intact long after the coding period of Google Summer of Code is over. It would also ensure that future contributors can work on adding support for many languages as MetaCall Core extends the functionalities for more languages on the upstream.
Check out the documentation website on metacall-jupyter-kernel.readthedocs.io.
- #4: Add Jupyter kernel setup
- #5: GitHub Action for CI Build
- #6: Documentation website
- #8: Linting with Flake8 and Black
- #10: MetaCall launcher for Jupyter kernel
- #12: Fixing Node execution with MetaCall
- #14: Adding automated language check with
- #16: Custom log message display for unsupported languages
- #18: Add magic support
- #20: Add support for shell commands
- #22: Fixing user experience features
- #23: Adding examples + GH actions + fixing magics bug
- #37: Embeding MetaCall REPL on the kernel
- #47: Adding red font for error output
- #50: Adding support for
- #51: Adding support for
- #54: Adding support for
- #56: Adding support for
- #60: Adding support for
$available, fixing Dockerfile and adding help links
- #62: Implement history access for the kernel
- #64: Adding examples for the kernel usage
- #68: Adding unit-tests for the kernel
- #73: Adding documentation
Bug Fix :
Maintenance, testing and updates:
- Total PRs sent: 24
- Total issues opened: 20
- Total commits: 49
- Total lines of code added: 4372
- Total lines of code deleted: 1664
- Total lines changed: 6036
It was a great experience working with the MetaCall team for GSoC’21. Thanks to my mentor Gil Arasa Verge and Vicente Eduardo Ferrer Garcia for giving me this wonderful opportunity and mentoring me. I look forward to continuing being a part of the MetaCall community and continue contributing to Jupyter kernel as well as other projects, like Core, along with helping new contributors find their way around the community.
Harsh Bardhan Mishra (@harshcasper).
Google Summer of Code 2021 Student from MetaCall.