Containers Make it Easy for Anyone to Use & Build Holochain Apps
Holochain Dev Pulse 02
Summary
We’re pretty excited about containers for many, many reasons but especially because containers help empower anyone to build on Holochain and use Holochain apps. Containers also eliminate the hassle of juggling individual instances and manual management of ports by hosting all apps and consolidating all communications into one set of ports. Basically, you can have a container for anything, to run on any platform, on any Holochain app and have everything communicate peer-to-peer.
One question for you: If you’re building on Holochain, what else do you need to know about Containers and JsonStrings? Let us know so we can let *you* know!
Holochain Dev Pulse Highlights:
1. Substantial progress made on Containers
2. Ribosome error handling refactored
3. Big efforts making JSON serialization more robust
4. Nodejs bindings for Holochain-rust now listed in npm
5. Tool built to notify team about stale PRs
6. Windows build automation working for Holochain Rust
7. Clarified DevCamp scope and registration for DevCamp #2 now open
Holochain Dev Pulse Highlights (Detailed)
1. Substantial progress was made on Containers
A. Architecting and syncing with Holo
B. Container configuration spec’ed
C. Rust implementation started
Holochain instances run inside Containers. We have written about containers recently, but here’s a summary. If the Holochain core library is an engine, the container is the car around it — body, fuel tank, transmission, steering, tires, etc. It takes the raw power of the engine and turns it into something we can use.
At the very least, a container implements an interface for the agent, a storage mechanism, and a way of communicating with the agent’s peers. When we say ‘interface,’ it depends on what sort of agent is interacting with the Holochain core. An end-user would want a container with a graphical interface, while a developer would want a scriptable interface to run tests.
The useful car analogy gets a little muddy, though, when we learn that a container is also meant to run and coordinate between multiple app instances at one time, providing a single interface and networking layer for them all. That’s more like if your car had a compartment that let you drop in as many engines as you liked, each of which could perform some distinct and exciting task! With Holochain-proto, each app ran in its own instance, with its own gossip and HTTP ports. If you’ve tried building or running apps on Holochain-proto, you know how much of a hassle it could be to juggle all those ports, especially if your firewall is feeling cranky! The container eliminates this problem by hosting all those apps and consolidating all communications into one set of ports.
Right now we have a few different container implementations. There is some overlap among them, so we’ve started consolidating features into one new core container, written in Rust. This container connects to the Rust container (which uses the Rust library) from the Holosqape Qt based GUI, using a UNIX domain sockets interface. Both of those containers also had a websocket interface, which is an example of the sort of overlap we’re trying to eliminate.
Hcshell has a JavaScript runtime that app developers can use to run tests against their DNA. This will likely be retired in favor of the NodeJS binding and container (exciting news about that below!). Also, we have built a container on Holochain-nodejs with an HTTP interface.
We’ve also started speccing out a configuration file format that will be used by all the official Containers created by the Holochain dev team, so that agent and app configuration can be transferred easily from one container to another. It defines details such as the location of the agent’s key file, their preferred storage interface, and all their app instances. This file will be invisible to the user, is really the thing to just make it all work.
Part of the reason for this work on Containers has been to support the work of the Holo Host team and enable the functionality that they will need to make Holochain apps work seamlessly on Holo.
2. Ribosome error handling refactored
Something always goes wrong when you are developing. You may have a syntax error, or your approach to a problem might be wrong, or your pet ferret might chew through your network cable. If your development stack isn’t giving you useful feedback about what’s going wrong, it can be incredibly frustrating to debug.
Therefore, we’ve been focusing on improving the error system and the helpfulness of the errors you receive. This will be a big relief to many of the developers who have started with Holochain-proto.
3. Big efforts making JSON serialization more robust
We are using the JSON serialization format to pass data between WASM and Holochain and to storage backends. Until now, we have been tracking how data has been serialized and deserialized manually. This worked to a point, but the growth of and co-ordination between the various Rust crates made manual tracking increasingly unwieldy. It was becoming common to discover that something had been “double serialized” somewhere between sub systems, and few debugging tools were available.
Zomes were starting to accumulate various JSON macros, dummy structs holding intermediate data and other unwanted boilerplate items. Zome functions all needed to be defined as accepting only String values as arguments, leading to awkward function signatures like, `fn foo(entry: String),` instead of `fn foo(entry: Entry).` Other minor annoyances, like handling null terminated strings consistently across WASM function calls, etc. were starting to pile up.
Our solution is simple. We added a new struct/type called JsonString that represents JSON serialized data. JsonString cannot be (de)serialized directly. Instead, the native rust From<Foo> and TryFrom<JsonString> traits should be implemented for data that needs to be serialized. We created a custom derivation macro, DefaultJson, that makes this easy for anything happening with the default serde behavior.
Core functions that straddle a (de)serialization boundary across subsystems now accept anything that can be converted to a JsonString through the native Rust `.into()` function. It is now impossible to double serialize data without explicitly writing logic to “take the safety off.” It is also impossible to forget to serialize data at all. Because serialization is now automatic, most (de)serialization boilerplate code has been removed from the Rust-HDK. Zome API function signatures can now accept anything that can be converted to JsonString natively. All Holochain core data that is exposed to zome logic, e.g. entries, addresses, links, etc. has been implemented the automatic conversions to/from JsonString. If you find anything that doesn’t convert cleanly to JsonString, it’s a bug! Let us know so we can clean it up!
This also provides future benefits as HDKs are implemented in new languages. Before JsonString was implemented, all (de)serialization logic was completely ad hoc. For example, an error might be represented as `{“error”: …}` in one part of the code, and `{“Err”: … }` in another. These minor inconsistencies can be worked around or cleaned up by hand on a small scale, but quickly become complex bugs when different systems need to interoperate. Even worse, this was all exposed to zome authors. Correct (de)serialization of core data types, like entries, were left in the hands of individual dApp developers!
With JsonString implemented, the Rust-type system enforces that there is only one way to serialize each data structure. Zome authors still need to define (or automatically generate) serialization logic for custom data types, but all native data serialization is now standardized. The process of implementing custom serialization logic should be much cleaner overall.
We will be building on top of this system in the coming weeks to audit and further standardize the core data structures. The upcoming audit will improve the long-term stability of the zome authorship API making it easier to write both zomes and HDKs.
4. Nodejs bindings for Holochain-rust now listed in npm
NodeJS bindings/container for Holochain-rust is now listed in NPM and can be installed via `npm install @holochain/holchain-nodejs`
Exciting news for NodeJS developers! We’ve published our Holochain-nodejs bindings and container to the NPM package repository. You can now download it and start hacking on your apps from Node! The fact that you can list Holochain as a dependency within some NodeJS project means that you can now access your Holochain app’s DNA from your Node app. You can do all sorts of things with this. Here are a few possibilities:
- Write tests for your zome functions in your favorite JS testing framework.
- Build a web server that represents one agent’s view into the app’s network.
- Bundle your app into an Electron desktop app.
5. Tool built to notify team about stale PRs
We’ve built a Mattermost chat bot that reminds our team to take a look at pull requests that have been open for a while. We found that some outstanding PRs were getting behind, so this will help us stay on track. It’s a nice system, and will be a great process for working with external contributors. We’re currently testing it and incorporating it into our development workflow.
6. Windows build automation working for Holochain Rust
We’ve been using a Continuous Integration pipeline for a while now, giving us automated testing and builds for Ubuntu Linux. We’re proud to say we’ve successfully added a Windows target to this pipeline. This is a big improvement; we have a few team members who run Windows and are looking forward to using some of our internal apps. The only remaining desktop system now is macOS.
7. Clarified DevCamp scope and registration for DevCamp #2 now open
Our dev team worked with the DevCamp team to incorporate previous feedback, and helped design the upcoming camps with our long-term plans in mind.
DevCamp registration is now open for purchase to the general public in order to more broadly boost development on Holochain. You can find all the details on the DevCamp site. The next DevCamp is from Thursday, November 29th to Monday, December 3rd.
Holochain Book Links
HDK: https://holochain.github.io/rust-api/0.0.1/hdk/
Zomes: https://holochain.github.io/rust-api/0.0.1/hdk/macro.define_zome.html
Rust: https://holochain.github.io/holochain-rust/
Contribute to Documentation
There is a deep commitment on the Holochain dev team to cultivate a culture of contributions and to make it relatively easy to participate. We would like to invite developers to contribute to the documentation. Therefore, we added some “suggest an edit” buttons to make it easier to contribute to certain sections of the documentation.
If you are interested in contributing to the code, a good starting point is to help with documentation. We feel it’s the best way to get a good understanding of Holochain’s architecture and codebase, and contribute to our project at the same time.
Developer Questions?
You can also find answers to many frequently asked questions.
Connect with the developer community. We want your feedback.
Like the content? Share it on Twitter and discuss it on our various channels (Mattermost, Telegram, Reddit, etc). Think we can improve on the content? Let us know. We’re open to content requests and suggestions — you can post them on our Mattermost chat, Holochain / Tech Q&A channel.
