Carp — New Cardano SQL indexer & replacement for db-sync

Carp is a new indexer for Cardano that saves its data to a Postgres database (SQL). If you’re not a programmer and don’t know what this means, we’ll briefly introduce this to you. If you’re a developer, feel free to skip the first section or skip directly to the developer documentation here.

What is an indexer?

Blockchains contain a lot of information — often hundreds of gigabytes worth of data. Applications like dApps and wallets need to quickly filter through this data to provide the fast user experience that users expect. This is what indexers are meant to solve. There are many different kinds of indexers and the best one for the job tends to depend on what you’re trying to do. SQL databases are a popular kind of indexer because they provide a lot of flexibility to users to query what they want while maintaining decent performance.

What indexers are available for Cardano?

Although recently there are many new tools to write non-SQL indexers in Cardano such as Oura and Scrolls , nearly all projects use an SQL database-powered indexer called db-sync. It’s a project started by IOG a few years ago and saves all the Cardano blockchain data in one large database so that nearly everything you could want is available to you.

So why is Carp revolutionary?

First, we’ll explain the core pillars of Carp and then see how this is totally different from what has come before it.

Carp has the following core pillars:

• Flexible: allow wallets and dapps full freedom in how they handle data
• Modular: it should only index what you need
• Speed: queries should be fast
• Type safe: developers should know exactly what data to expect to avoid any critical issues in their products.
• Documented: it should be easy for any developer to get started with Carp

How is Carp more flexible than other solutions?

To understand the importance of flexibility, first, let’s talk about wallets for Cardano. The first-ever Cardano wallet created is called Daedalus. One of the main complaints about Daedalus in the early days is how long it would take for new updates to come out. In contrast, other newer wallets like Eternl, Flint, and Nami update much faster with smaller teams.

So why was Daedalus so slow to release new versions? The ‘why’ is because Daedalus itself is actually just a user interface. All the data processing is done by a separate application (managed by a separate team) called cardano-wallet. What this means is that if the Daedalus team wanted to add a new feature, they would first have to make a feature request to the cardano-wallet team, wait for it to be added, then integrate it into Daedalus. Since human communication is slow and often incorrect, this back-and-forth was often an expensive and time-consuming effort.

In contrast, newer wallets in Cardano nearly all use a library I started called Cardano Multiplatform Lib (CML — previously known as CSL (Cardano Serialization Lib)). This library makes all the core wallet functionality (creating addresses, creating transactions, etc.) available directly to wallets instead of having them require calling another service. The library implements the Cardano binary specification directly, so wallets have full flexibility to do exactly what they need without having to ask for new functionality to be added.

db-sync is the Daedalus equivalent of indexers. It returns data in a very opinionated format, and if it doesn’t fit your use case it’s not so easy to modify. For example, if you ask db-sync for information about a transaction, instead of giving you the full transaction, it will instead give you bits and pieces of information. This is nice if all you cared about were those pieces, but not so useful if you wanted the full information.

Carp provides you with the full data. Instead of providing you bits and pieces, we instead give you the full binary data of the transaction. Thanks to our work on CML, almost all wallets and dApps can easily parse this binary data to get exactly what they need instead of the database needing to be opinionated.

Comparison of SQL definition of the transaction table for Carp vs db-sync

How is Carp more modular than other solutions?

Many indexers suffer from something called “feature-creep,” where more and more features get added over-time causing the application to end up being bloated and slow. For example, db-sync indexes all delegations on-chain in a special table and yet many applications don’t actually need this information.

Carp, in contrast, is fully modular. Anybody can create their own modules and combine modules written by others. This allows you to easily create an indexer that specifically meets your use case! We do this by introducing three concepts:

1. Carp tasks. Tasks are like mini-indexers that only index some specific kind of information (such as a task that only saves addresses it sees on-chain). Each task can specify parameters that change how they behave.
2. Execution plan. Carp allows you to specify which tasks you want to use for your indexer. Your execution plan can specify which tasks you want to use and which settings you want to use for which tasks.

An example of an execution plan specifying 4 tasks and parameters for them. They are meant to be human-readable

3. Execution graph. Once you’ve defined your Carp tasks and your execution plan, Carp will generate for you an execution graph that runs all your tasks in parallel to maximize throughput

An example execution graph for Flint that keeps track of transactions, staking credentials, and NFTs

How is Carp faster than other solutions?

Thanks to the fact that you’re only indexing what you need and you can easily write a custom indexer to get the data in the exact format you need it, Carp allows you to write faster applications!

For example, getting the transaction history for a user takes under 100ms in the usual case for our prototype integration in Flint! Depending on the query, this is a 10x improvement

How is Carp safer than other solutions?

One of the biggest sources of bugs in computer programs is when two different applications need to talk to each other. Often, teams miscommunicate leading to subtle bugs. To avoid the same issue with Carp, we provide “type definitions” for Carp as a Javascript package so that any application that uses Carp can make sure they are processing data in exactly the format Carp needs and will get a warning if they do things incorrectly.

How is Carp better documented than other solutions?

If you’ve read this far, then maybe you’re interested in trying Carp yourself! We spent a lot of time making very developer-friendly documentation so that it’s as easy as possible for new projects to launch using Carp. You can find the documentation here. You can also find the full code on Github here.

• Okay, to be fair, all the indexers in Cardano are fairly well documented, so on this point it’s hard to argue that we’re “better”, but we did put a lot of work into the documentation!

What’s next for Carp?

We’ll continue updating Carp as needed and next, we’ll work on implementing Carp into Flint Wallet. However, what comes after that is partially up to you! Any developer can write Carp tasks and contribute to Carp (fully open-source MIT license). Additionally, Carp was funded through a one-time Catalyst proposal for $35,000 (definitely less than the cost to build this kind of tool, let alone maintain it! db-sync is well into the 6-digits in project cost), so if you want dcSpark to continue development, don’t hesitate to ask us for follow-up Catalyst proposals for continued work.

Acknowledgments

Thank you to Catalyst voters who supported our Catalyst proposal to build Carp and other tools that made Carp possible such as our work on Flint Wallet and Cardano Multiplatform Lib.