Bitcoin Event Database

Opening up bitcoin event database endpoint to the public

Today we are opening up our bitcoin event database API to the public.

Now you can directly crawl the same timestamped event database that powers Bitsocket. This means you not only can subscribe to incoming events via SSE (server sent events), but you can also crawl the entire event history up to 24 hours. You’ll never miss a relevant event again.

How is this related to Bitsocket?

The Bitsocket SSE endpoint is internally powered by an event database which stores deserialized bitcoin transactions as events.

Until today, this event database has been only used internally for powering Bitsocket and didn’t have a public API. But now we are opening up the event database itself as an API so anyone can crawl it, just like you can Bitbus.

Since it’s powered by the same DB as Bitsocket, it’s served from the same Bitsocket domain, just with a different path. Instead of the usual /s endpoint for Bitsocket, you can simply make a POST request to the path:/crawl.

You can access the crawl endpoint at the following paths:

POST https://txo.bitsocket.network/crawl
POST https://bob.bitsocket.network/crawl

What does the event DB let me do?

The event database API lets you filter and crawl the last 24 hours worth of realtime events from the Bitcoin peer network using Bitquery:

bitquery

The Bitsocket realtime SSE (server sent events) API is simply an application built on top of this event DB which turns the event database into a realtime SSE endpoint.

While Bitsocket SSE is very useful for filtering realtime transaction events on the Bitcoin network, the limitation is that you can only start receiving the events once you connect to Bitsocket. There is no way to “crawl” the past events.

And this is what we are opening up today. You now have direct access to the entire event database history (up to 24 hours) through an HTTP API.

How is the event DB different?

So what exactly is this “Bitcoin event database”? To understand the difference it’s best to try to categorize the bitcoin database APIs we provide as of today:

1. Block database: Index all the confirmed transactions on the blockchain. All transactions are timestamped per block (using block height and block hash).
2. Event database: Index realtime transaction as events. All transactions are timestamped using the real world timestamp of when the transaction was first seen.

1. Block Database

A block database stores confirmed transactions. All block databases have the field blk, which looks something like this:

{
  ...
  "blk": {
    "i": <block height (index)>,
    "h": <block hash>,
    "t": <block time (the unix timestamp of when the block was mined)>
  }
}

Most users of Planaria APIs should be aware of this syntax. Because everything is indexed with the blk attribute, you can easily filter and sort by block height. All transactions within a single block have the same blk attribute.

A “block database” is updated every time a new block is discovered.

2. Event Database

An “event database” is all about realtime. It directly listens to the Bitcoin peer network for every single new transaction and stores them as events. The key difference here is that every “event” is timestamped with a real world unix timestamp.

{
  ...
  "_id": <unique id which increments over time>,
  "timestamp": <unix timestamp of when the transaction was discovered>
}

Using this novel scheme we can implement realtime SSE (Server Sent Events) APIs such as Bitsocket.

How to crawl the event DB

To crawl the event database, you simply make a POST request to the /crawl endpoint with a Bitquery which describes what you are looking for.

The endpoints are at:

• https://txo.bitsocket.network/crawl
• https://bob.bitsocket.network/crawl

Note: the crawl endpoint is powered by the same engine that powers Bitbus, so they work exactly the same. The only difference is this API crawls the event database instead of a block database.

You can use any programming language to crawl the bitcoin event database. Here’s a JavaScript example which fetches all OP_RETURN transaction events from 100 seconds ago.

fetch("https://txo.bitsocket.network/crawl", {
  method: "post",
  headers: {
    'Content-type': 'application/json; charset=utf-8',
    'token': <your planaria token from https://token.planaria.network>
  },
  body: JSON.stringify({
    "q": {
      "find": {
        "out.o1": "OP_RETURN",
        "timestamp": {
          "$gt": Date.now() - 100000
        }
      }
    }
  })
}).then((res) => {
  // transform to text format
  return res.text()
}).then((res) => {
  // the response is in NDJSON (New line separated JSON) format, so split with newline.
  res.split("\n").forEach((line) => {
    document.write("<pre>" + line + "</pre>")
  })
})

Just like Bitbus, you need to use Planaria token to make requests to the crawl endpoint.

BitFS Integration

There’s one more thing. Today’s rollout is not just about the event database, but also an update to Bitsocket as well. Now Bitsocket is integrated with BitFS.

With this release, the event database also starts indexing f attributes, which is a new way to store large data chunks in a bitcoin transaction. The f attributes represent BitFS URIs for push data chunks larger than 512 bytes.

The l attributes which were used to represent large push data have been deprecated because it no longer makes sense to store potentially gigabytes of data in a DB. Instead we store them in a file system:

4. How does bitFS store files?

Here’s an example (take a look at out[0].f3):

{
  ...
  "out": [
    {
      "i": 0,
      "e": {
        "v": 0,
        "i": 0,
        "a": "false"
      },
      "len": 33,
      "o0": "OP_0",
      "o1": "OP_RETURN",
      "s2": "19HxigV4QyBv3tHpQVcUEQyq1pzZVdoAut",
      "b2": "MTlIeGlnVjRReUJ2M3RIcFFWY1VFUXlxMXB6WlZkb0F1dA==",
      "h2": "31394878696756345179427633744870515663554551797131707a5a56646f417574",
      "f3": "e3e1420283717d82d83b4166f133d55350e8ee634dd96bfbf1ae808419e35bd7.out.0.3",
      "s4": "image/jpeg",
      "b4": "aW1hZ2UvanBlZw==",
  ...
}

For example if you receive the above event, and would like to process the out[0].f3, you can simply make an additional fetch request to BitFS:

fetch("https://x.bitfs.network/" + out[0].f3).then((res) => {
  // do something with res
})

This BitFS integration means:

1. Lean: Because every attribute larger than 512 bytes gets stored on BitFS instead of on the DB, Bitsocket events are always lean and waste less traffic.
2. Extractible: You can easily look up the corresponding raw push data using the f3 attribute on Bitfs since the BitFS urls are deterministic.
3. Consistent: Now the Bitsocket endpoints and Bitbus endpoints have the same consistent schema. They both use the f attributes to reference large data chunks using Bitfs.

You can learn more about this here:

Bitsocket

Usage with Bitsocket Realtime Endpoint

The cool part about this new API is that it’s powered by the exact same event database that powers Bitsocket, so the data will be exactly the same.

So for example, you can do an initial synchronization using the https://txo.bitsocket.network/crawl endpoint first and simultaneously start listening to the https://txo.bitsocket.network/s SSE endpoint.

You won’t have to worry about missing an event because they both come from the same event database. Even if you miss an event through the realtime SSE Bitsocket connection, you can always “crawl” the relevant past events using the crawl endpoint.

Conclusion

We recently did an internal hack week where we used our own APIs to build apps and make sense of various datasets on the blockchain.

We did this in order to understand how the API can be used so we can understand various pain points and design the API better going forward.

And through the experience we realized we were missing one important feature: A way to query the realtime event database. While the realtime Bitsocket SSE endpoint is useful, it only lets you subscribe to the events AFTER you make the connection. There is no way to know what happened before you make a connection. This is critical when you load up an app for the first time.

Now with the full access to the event database, the suite is complete:

1. Crawl block data: Use /crawl endpoint of Bitbus
2. Crawl event history: Use the /crawl endpoint of Bitsocket
3. Listen to realtime events: Use the /s endpoint of Bitsocket

This should be all you need to build and experiment with all kinds of flexible apps and tools that make use of on-chain bitcoin data.