Planaria Systems Programming Interface

Opening up a whole new dimension to programming Bitcoin powered state machines

Planaria is an application framework and a network that lets you easily build and deploy Bitcoin powered application backends, as well as open them up as public utility to let other developers build serverless Bitcoin applications on top of (and eventually generate revenue) without having to run their own node.

  1. CRUD API: Planaria listens to realtime Bitcoin transactions and creates an autonomous state machine based on the transform code you write, which utilizes Bitcoin transactions as database commands, Creating/Updating/Deleting custom database entries through Bitcoin transactions.
  2. HTTP + SSE API: A DB is meaningless if the outside world can’t query it. Planaria also provides an out-of-the-box, 100% open standards based, HTTP + Server Sent Events API, which can be used to build any app (Web app, mobile app, server-side apps, etc.)

And today, Planaria is opening up a whole new layer of its programming interface. On top of “application programming” you’ll be able to do “systems programming”.

You will be able to run all kinds of things on Bitcoin. Some examples:

All this is now possible because Planaria is opening up its “systems programming interface” to let developers do more low level things while taking full advantage of the exact same transparent computing model of Planaria, where each Bitcoin transaction is a command. This means all commands and their resulting state transitions are 100% transparent to the world, WITHOUT having to store the entire state machine on chain.

Instead of storing the entire state machine on chain, you can basically “reproduce” a deterministic state machine from Bitcoin transactions, with minimal data stored on Bitcoin itself.

If you’re not technical but interested in brief explanations on how all this will happen, scroll to the bottom of the post.

Otherwise, let’s go!

Introducing Planaria Systems Programming Interface

Previously Planaria provided an “Application Programming Interface”, driven by Bitcoin transactions to perform CRUD actions on the built-in MongoDB instance. While powerful, it’s just one specific type of state storage. We can go further.

Today Planaria is taking the first step to opening up its “Systems Programming Interface”.

Planaria node developers now have access to much more powerful features, including:

  1. Full access to the file system
  2. Custom API endpoints, as many as you want.
For backend developers, this means you can now build much more powerful state machines and API endpoints powered by Bitcoin, using Planaria.
For app developers, this means you will soon have access to these powerful new API endpoints built by the aforementioned Planaria node developers who publish the node to the Planaria Network.

1. Planaria File System

Recently Planaria added a new feature called “Planaria File Serve API” which provides an instant static file server feature. Basically it let you write to a folder named assets and made them instantly available as static files through a new HTTP endpoint.

This simple feature proved to be very powerful, powering interesting applications like Bitstagram.

Today we’re taking this to the next level.

If the file serve API gave you limited access to a single folder named assets, the new Planaria File System interface lets you access the ENTIRE file system in which Planaria (for writing) and Planarium (for reading) operate.

  1. The Entire File System: In addition to the built-in MongoDB instance which acts as the backbone of Planaria, you can implement additional state storage by writing to and reading from files.
  2. Flexible: Unlike the file serve API which was limited to a single folder named assets and was public by default, the new file system interface is private by default, just like how a regular backend works. This means you can build a transparent backend without having to make everything public.
  3. Build your own file serve API: The flexibility allows for various models of backend privacy models. For starters, the new file system interface can be used to build as many file serve apis as you want. And it’s much more powerful because you can write your own custom file serve handler (The old file serve API was simple but not customizable).
  4. Private File System: Often you don’t want your files to be immediately accessible to public HTTP access. For example, you may be storing private keys. You may be storing 3rd party API keys that should not be exposed to the public. (Similar to how programmers never commit environment variables to version control systems like git) Or you may be storing a file-based-database, which exists as a blob of file and is meant to be accessed through a database query interface instead of accessing the raw file itself. So on and so on. By providing a private file system, you can build extremely flexible applications that still take full advantage of Bitcoin’s transparency (since all “commands” and “events” are Bitcoin transactions)

2. Planarium Custom Endpoints

Until now, every Planaria microservice had two powerful built-in API endpoints:

  1. Bitquery: An HTTP endpoint (located at /q) for querying the blockchain using a Turing complete portable query language written in JSON, also named “Bitquery”.
  2. Bitsocket: A server-sent-events endpoint for programmable push notifications API (located at /s) that lets you listen to very specific patterns of Bitcoin transactions using the same Bitquery query language.

Thanks to the expressive nature of the Bitquery query language, developers have already been able to make all kinds of powerful queries into the Bitcoin blockchain just with the two APIs.

But now with the file system support, you may want to add additional endpoints to serve your files, or even other data added to your Planaria state machine through other means.

And starting today, you can add your OWN custom GET HTTP endpoints through planarium.js

Note: Only GET requests are allowed because ONLY Bitcoin can write to Planaria. If you want to write to Planaria, you don’t make a POST or PUT request to Planaria node itself. Instead you “post” by making a Bitcoin transaction.

Here’s what it looks like:

Internally, the Planarium API endpoint is powered by express.js, and the routes are directly mapped to the native GET handlers for express.js.

In this example we’re defining a GET /c/:id endpoint which will automatically map to the following URL:

https://[HOST_PATH]/[PLANARIA_MACHINE_ADDRESS]/c/[id]
Note: You can add as many route handlers as you want!

Here’s an actual example of what this may look like:

https://data.bitdb.network/1KuUr2pSJDao97XM8Jsq8zwLS6W1WtFfLg/c/....

All Planaria endpoint URLs are prefixed by its machine address. Each machine is uniquely identified by its Bitcoin address (instead of its host HTTP URL). This is how Planaria ensures application portability.

Because the machines are identified by the Bitcoin address and NOT the host’s HTTP URL, all Planaria state machines are independent from whichever host they’re being served from, and independent from the server-client based HTTP scheme. It’s Bitcoin native.

The same machine, when run by another host, would look something like this, for example:

https://files.bitdb.network/1KuUr2pSJDao97XM8Jsq8zwLS6W1WtFfLg/c/....

They would have EXACTLY the same state as the other endpoint because they’re constructed from the same code.

3. The Initializer

With all these new things you can do with Planaria, you’ll immediately realize you need some way of initializing various modules you’ll be using throughout the rest of the machine’s lifecycle. For example, you may want to:

  1. Initialize a file based database once when a Planaria machine is set up for the first time.
  2. Create empty folders that will be used throughout the rest of the machine lifecycle.
  3. Initialize certain libraries that will be used throughout the rest of the machine life.
  4. Import an existing AI model exactly once as the “genesis” mental model.

To facilitate this, Planaria has added a new event handler named oncreate. The oncreate event will be called EXACTLY once when the machine is initialized.

And of course, since there are two containers, we have oncreate initializer for both:

  • planaria.js: Located under oncreate. The Planaria container is for writing state, so the oncreate gets executed EXACTLY once when the machine runs for the first time.
  • planarium.js: Located under query.api.oncreate. It’s slightly different for Planarium. The Planarium container is for reading only and is stateless. Therefore oncreate for Planarium gets executed every time the container is created.

Here’s an example planarium.js setting (under query.api.oncreate)

Here’s an example planaria.js setting (under oncreate)


Example

Let’s take a look at how all these features come together using a real world example: C://, a content addressable file system over Bitcoin.

C:// generates filenames by SHA256-hashing the content uploaded through Bitcoin and storing the file content under the content hash as filename. This means there’s exactly one universal filename per unique content, which introduces interesting properties that lets us use the files as building blocks for deterministic applications.

But first before we go in, you can find the entire code AND the public node running at:

1. Initializer

For this machine we make use of the newly introduced event handler called oncreate.

For Planaria, this gets executed exactly once when the machine first starts its life. It is run only once the first time the machine is run, and after that onrestart is called instead whenever the node restarts.

Since we only want to mkdir once in the beginning, this belongs inside oncreate of planaria.js.

module.exports = {
...
oncreate: async function(m) {
// create C:// folder
await mkdir(m.fs.path + "/c")
// create LMDB folder
await mkdir(m.fs.path + "/lmdb")
// Initialize LMDB
en.open({
path: fspath + "/lmdb",
mapSize: 2*1024*1024*1024,
maxDbs: 3
});
db = en.openDbi({ name: "mediatype", create: true })
},
...
}

We also want to access the files and the LMDB instance from planarium.js since we want to provide an API endpoint. So we initialize them inside planarium.js as well:

module.exports = {
query: {
api: {
oncreate: async function(m) {
dbpath = m.fs.path + "/lmdb"
await mkdir(dbpath)
filepath = m.fs.path + "/c/"
env.open({ path: dbpath, mapSize: 2*1024*1024*1024, maxDbs: 3 });
db = env.openDbi({ name: "mediatype", create: true })
},
Remember, the oncreate for Planarium gets executed every time it starts because Planarium is just a stateless HTTP/SSE server that consumes from the actual state machine which is Planaria.
On the other hand Planaria’s oncreate gets executed ONLY ONCE in its lifetime, the first time the Planaria state machine is created.

2. Write Files to Custom Path

You may have noticed the m.fs.path variable used in the code above. This environment variable contains the root path of the Planaria file system, shared by both Planaria and Planarium containers.

Internally, Planaria creates a folder structure that looks like this:

/fs
/[Address for Planaria Machine1]
/[Address for Planaria Machine2]
/[Address for Planaria Machine3]

Note that this example is an example of a heavy use case, and in many cases you would be running just a single machine and there will be only one folder under /fs.

But essentially, each Planaria machine has its own sandboxed filesystem. This is important because a Planaria node can run multiple machines simultaneously, and files from each machine shouldn’t spill over to another file system.

That said, you don’t have to know this deep to make use of the API. All you need to remember is that the m.fs.path that gets passed in from every callback onmempool, onblock, oncreate, onrestart references the root folder of the machine’s file system. You simply need to use that variable.

For example if you want to create a folder named /c you simply need to do:

mkdir(m.fs.path + "/c")

Another important thing: The file system is shared across Planaria and Planarium containers. So you can WRITE state from Planaria, and make use of the state from Planarium.

Here’s a more concrete example usage of this m.fs.path variable:

module.exports = {
onmempool: function(m) {
// create a folder named /c/[hash] under the root file system
fs.writeFile(m.fs.path + "/c/" + hash, buf, function(er) {
...
})
...
}
}

3. LMDB

For a file server to work properly, we need to set the content type header for every file served. Otherwise browsers may not understand how to render and just download the files to the file system instead of displaying them in the browser.

But extracting this metadata at file load time may be inefficient. So here we will try to store content types at crawl time, cache it once, and then use it for every file load.

One way to do this would be to query the built-in MongoDB instance which already stores this metadata. But we’ll try something different here because we don’t want to spam the core database every time a file is requested.

Let’s use a secondary, file-based database. In this case we use LMDB (Lightning Memory-Mapped Database). LMDB is a lightning fast key/value database that’s optimized for read operations, is very reliable thanks to its transaction based nature, and allows concurrent access from multiple processes, which is perfect for what we’re trying to achieve here — storing { hash: content-type } key/value pairs.

  1. We create the database under m.fs.path + "/lmdb" folder from planaria.js.
  2. Open the database
  3. Then use the database connection to “put” the hash:type key value pair for every onmempool event.
const lmdb = require('node-lmdb')
var en = new lmdb.Env()
var dbi
module.exports = {
...
oncreate: async function(m) {
...
await mkdir(m.fs.path + "/lmdb")
en.open({
path: m.fs.path + "/lmdb",
mapSize: 2*1024*1024*1024,
maxDbs: 3
});
db = en.openDbi({ name: "mediatype", create: true })
},
onmempool: async function(m) {
let txn = en.beginTxn()
txn.putString(db, hash, type)
txn.commit();
}
}

Once we’ve created these entries, we can then access them from planarium.js and serve all this. We just need to add a new custom API endpoint, which I will explain in the next section.

4. Custom Route

You can define your own custom external facing HTTP Routes under query.api.routes inside planarium.js.

Remember that ONLY Bitcoin can write to Planaria. There is no “POST” or “PUT” or “DELETE” requests to Planaria. All writes to Planaria are done through Bitcoin, so if you want to write, you make a Bitcoin transaction.

Therefore the routes interface assumes only GET requests, which is directly bound to the get handler for express.js (Planarium API is powered by express.js)

const level = require('level')
const db = level('mediatype')
...
module.exports = {
query: {
api: {
routes: {
"/c/:id": function(req, res) {
// 1. Generate filename
let filename = filepath + req.params.id;
// 2. Get the content-type info for the hash from LMDB
let txn = en.beginTxn()
let value = txn.getString(db, req.params.id)
txn.commit()
if (value) { res.setHeader('Content-type', value) }
let filestream = fs.createReadStream(filename)
filestream.on("error", function(e) {
res.send("")
})
filestream.pipe(res)
},
...
}

Notice how we’re accessing the same LMDB instance we wrote to earlier (from planaria.js).

The two containers (planaria and planarium) share the same file system so you can write from planaria.js and read from planarium.js. Here we access key/value pair we stored earlier inside planaria.js, after which we set the value as the content type header before serving the file.

Feel free to learn more about how it works at:

Also you can use the API from the public node too:


What does this all mean?

Everything is a file.

If you think about it, everything runs on a file. And all applications are just files, which make use of other files.

Once you see it this way, you’ll realize that Planaria can be used to build all kinds of full fledged Bitcoin-powered computation backends.

Below I’ve listed some interesting technologies that are all built on files just to help with imagination.

But think about it. EVERYTHING IS A FILE.

Everything is Bitcoin.
  • Flat File Database: You now have access to file-based databases such as LMDB.
  • Keys: Thanks to the default privacy model, you can even store Bitcoin keypairs, RSA keypairs, 3rd party API keys, or whatever key system you want to store in your backend, WITHOUT exposing them to public. This can be used for signing/verifying or encrypting/decrypting incoming data, or interacting with 3rd party APIs.
  • Bitcoin Transactions: You can even store unsigned Bitcoin transaction templates and use them in your computations and APIs. Stop here and dwell on this a bit.
  • Version Control: Git is a file based version control system. Now you can store and update git (or similar systems) on Planaria.
  • Artificial Intelligence Training Models: You can store machine learning models on a Planaria node and train them through Bitcoin transactions. This could either start with an imported training model, or could start from scratch.
  • 3D Printing model files: Store and update the 3d printing model over Planaria. And at some point you can even “commit” the change by making another transaction that uploads the up-to-date file.
  • Code: You can even create a state machine that not only updates its own state, but also updates the very state transition logic itself, implementing an “upgradeable” state machine powered by Bitcoin transactions.

By the way, did you notice how every aspect of everything I’ve just explained is a money making opportunity?

You can run powerful deterministic transparent services powered by Bitcoin and make money. Today. People will pay the services to run programs. And Bitcoin will act as the immutable witness, in case any service provider tries to cheat, which provides “trust” for these operations.

That’s what Bitcoin is capable of. And that’s what Planaria does.

And best of all, Planaria is not some science fiction or scam whitepaper selling the future that will never come. You can go use public nodes right here:

Or even run your own and provide your own public node as service.

Planaria is here, today.

All of what’s explained in this article is possible, today.

Bitcoin can do all this, today.

Let’s go build.

Join us.