Motoko, the Internet Computer’s Native Language, Turns Two
Since launching, many IC projects have adopted Motoko for development. Here is an update on the language and where we’re at.
Happy second anniversary, Motoko!
Two years ago, we announced the launch of Motoko, an actor-based programming language designed for the Internet Computer. A lot has happened in the meantime — most notably, the Internet Computer finally underwent Genesis and launched.
Since then, many projects have adopted Motoko for development and are running on the Internet Computer, with some 400 repositories on GitHub. Projects and repos are becoming more technically complex, with dapps and open internet services employing different canisters where the core “business logic” is written in Motoko, but an associated data storage or token might be implemented in Rust. In that way, we’re seeing increasingly dynamic systems enabled by WebAssembly.
The Motoko team has also not been idle over the past year. We are constantly working on improving Motoko, both the language and its implementation, and on making it more easily available and accessible.
Opening up
The biggest milestone for the Motoko team was that the language and its libraries have finally been open-sourced. We had been working toward that goal for a long time, but various technical and organizational blockers (such as testing infrastructure) had to be addressed before it could finally happen. Motoko and its base library are now public on GitHub, including all sources and the full issue and commit history, open for external contributors.
Perhaps even more visibly, the Motoko Playground launched. By allowing users to deploy and test canisters through a simple web page, it vastly reduces the barrier to entry for playing with the Motoko language and the Internet Computer. We on the Motoko team find ourselves using the Playground all the time to try out simple things in a fully-featured Motoko+IC environment.
Vessel, the package manager for Motoko, has become a more central part of the Motoko ecosystem as well, and allows developers to easily pull libraries from third parties into their projects or make their own libraries available to other projects.
There have also been numerous improvements and additions to the Motoko documentation, though of course there is more to do. The documentation is now open source as well, and we welcome contributions from the community, either in the form of suggestions for improvement or actual material.
Memory
The most important project for us right now is to make Motoko’s use of memory more scalable. As you are aware, Motoko is a managed language, meaning that programmers are freed from worrying about the nitty-gritty and error-prone details of memory management. This is achieved by a garbage collector, a piece of code added by the Motoko compiler that discovers when data is no longer needed and the space it uses can be reclaimed. Garbage collectors are used in all high-level programming languages, and their state of the art has reached a level of sophistication that beats most manual attempts to manage memory.
That being said, Motoko started out with a very naive collector, because that was the fastest way to get us off the ground. It was a simple semi-space collector that keeps copying the live data from one half of the active heap to the other. But our early adopters are taking the IC to the limits of its memory capacities faster than we anticipated, so it was clear that we needed to improve quickly. After rewriting the entire runtime system from C to Rust, we now offer a compacting collector that uses much less space. We also drastically improved the scheduling of collections, so that they only occur when enough garbage is likely to have accumulated.
Now, we are progressing towards a page-based heap, which will allow localizing the collector’s work to individual IC memory pages, and thus reducing memory charges by dirtying only a few pages at a time. This is a prerequisite step for the next big improvements: a modern generational and ultimately incremental collector that minimizes and splits up garbage collection into small, consecutive work packages that can be dealt with evenly over time, avoiding unpredictable spikes in cycle usage.
Another important topic is stable memory. This is a separate memory region that the IC allows to survive canister upgrades. In the Motoko implementation, we use it to automatically back up the contents of a canister’s stable variables before doing an upgrade. Unfortunately, this can be costly. Ideally, one would like to store stable data in that memory region directly, but extending garbage collection to this would require direct access to that memory to be feasible, which WebAssembly and the IC do not yet support. As a stop-gap measure, we will soon provide an interface for accessing stable memory manually, with the hope that the community will build library abstractions such as a key/value store or even a database that reside in stable memory.
Safe Canister Upgrades
Another important topic for us is increasing the safety and reliability of canister upgrades. It was always the plan that dfx would be able to check that the interface of a new canister version is backwards compatible with the old one, in the sense that all existing clients depending on it and not (yet) being aware of the upgrade will continue to function.
We specifically designed Candid to define, in a checkable manner, what a safe upgrade is. This is expressed in terms of subtyping rules on Candid interfaces. Joachim recently explained these in a blog series. We are actually able to formally prove, as a form of machine-verified soundness statement, that these rules correctly prevent clients from breaking. But while this was planned from the beginning, we only now get to build the infrastructure into the IC, dfx, and the Motoko compiler to actually perform these checks.
A similar but separate issue occurs with stable variables. While these are not part of the public interface of a canister, they are a sort of private interface between two versions of a canister, and generally need checking in an analogous way. Concretely, the new canister should still be able to read the contents of existing stable variables, which requires restricting changes to their type across an upgrade, otherwise a canister might accidentally lose data. This check and the infrastructure for it is also currently being implemented by the team.
Language Features
Of course, we have also improved the Motoko language itself. We added quality of life features and consolidated some of the less felicitous aspects of the design.
For example, do? blocks make working with option types ?T much more convenient. You no longer need to switch on each option to access its value. Instead, inserting a single bang ! is enough, provided the entire computation is wrapped up in a do? block to encapsulate the error case on some outer level. In the future, we plan to generalize this feature to other types such as Result.
Another pain point with Motoko’s structural type system was that there was no “inheritance” that allowed defining object or variant types in an incremental manner, by extending previously defined types — instead, all fields had to be repeated. This has now been made unnecessary with the introduction of the new type-level operators and and or, which express ways to refine a given type with additional information, or even combine two types.
In terms of consolidation, for example, we refined the object and block syntax to be less ambiguous, and we removed the clumsy WordN types and the need for tedious conversions in favor of additional operators in the NatN and IntN families of types. We also made various improvements to type checking, especially reducing the need for spurious type annotations when using overloaded operators or literals. And last but not least, we tried to make compilation reproducible, i.e., to avoid dependencies on the environment, such as source file paths occurring in assertion error messages.
There are many other nice language features we have been thinking about, and will hopefully get to in the future. But we are currently focusing our efforts on consolidating the language and making it more scalable and safe, including some performance optimizations, such as for iterating over arrays.
Libraries
Last but not least, libraries are important, too. There have been various smaller additions to the Motoko base library over time, though obviously, there could always be more. As a benefit of open-sourcing, we are hoping for contributions from the community to make the Motoko library richer, or for third parties to provide their own, additional libraries by making them accessible through vessel.
The main goal for us right now is to expose some features of the IC through system libraries that cannot currently be accessed. For example, transferring cycles, directly accessing stable memory (with 64 bit address range), supporting method access checks, and perhaps the heartbeat. Some of these already exist in experimental form, but will require some more tweaks before we can officially release them to the world.
Finally, we are aware that it is currently difficult to test and debug Motoko programs, especially when they are supposed to interact with other canisters. Unfortunately, to a large part, that is a non-trivial problem due to the nature of the IC itself. For example, it obviously cannot allow setting breakpoints or stepping through code that is running live on the network, since that is incompatible with its underlying blockchain model, which does not support random interruptions of computations. So the general problem cannot really be solved on Motoko’s end. There are some ideas for providing mechanisms for observing the state or behavior of live programs, but that will take time to design and implement in the system. Until then, we would like to ease the situation to the degree we can by at least adding easier logging and perhaps offline debugging facilities to Motoko.
Community
Of course, a language is worth little without its community. We have been thrilled by the positive reactions we received for the language — as well as the constructive criticism and suggestions for improvements. Please bear with us if some things take time.
We would like to take this opportunity to invite the community to contribute to Motoko, its libraries, documentation, and ecosystem. Whether you participate in the forum, file issues or feature requests, create PRs, perhaps release your own libraries through vessel — everything is welcome that makes working with Motoko more productive, more convenient, more fun!
____
