Subsquid Now Supports Ethereum

Published in

SQD Blog

6 min readOct 26, 2022

After months of hard work by Subsquid’s core development team, we are proud to announce the launch of squids and our entire ETL (extract, transform, load) service platform on the Ethereum network. After a year of stable operations in the Polkadot ecosystem, providing data to hundreds of teams across dozens of parachains, we can’t wait to offer a better alternative for data retrieval and transformation to Ethereum developers.

In the interest of building the best possible data standard for Web3 builders, we are carrying out a beta program for the squids on Ethereum. To register for that program, please fill out the form here or contact our growth team on Telegram.

Indexing and ETL for next-gen dApps

Founded in late 2021, Subsquid is the team behind squids, a new standard for Web3 data extraction and transformation. The idea for Subsquid was born out of the need for customizable and performant indexing, which co-founder Dmitry Zhelezov faced when building the infrastructure for a decentralized video streaming service. None of the available tools at that time could handle the fast-pace of this next-gen use case.

When making updates to the dApp using existing solutions, the ETL infrastructure took ages to sync, and iterations added weeks to development time. Ultimately, Dmitry understood that a modular architecture was needed to speed things up — only by splitting on-chain data ingestion from transformation and presentation would it be possible for the dApp to get the data it needed and serve it to users with the required low-latency.

Subsquid’s architecture consists of Archives, which pre-index the data, and squids, where the developer’s custom logic is applied. Thanks to Dmitry’s novel design decision, Subsquid can sync up to 50,000 blocks per second. Apart from record breaking speed, squids also allow for a level of customization and developer freedom that just isn’t provided by other indexers and ETL frameworks.

Full control over the target database (Postgres), including custom migrations and ad-hoc queries in the handler.
Custom target databases and data formats (e.g. CSV).
Arbitrary code execution in the data handler.
Extension of the GraphQL API with arbitrary SQL.
Secret environment variables, allowing for seamless use of private third-party JSON-RPC endpoints and integration with external APIs.
API versioning and aliasing.
API caching.

To learn more about Subsquid’s architecture, visit the documentation.
Ready to start building? Join the beta!
Got a Subgraph to migrate from? We’ve got a guide for that.

Presently, Subsquid is used by every major project in the Polkadot ecosystem. After several updates and extensive battle testing, it is now time for the rest of Web3 to be transformed by squids!

What is a squid?

A squid is a faster, easier, and cheaper way to retrieve Web3 data, transform it, and use it to build advanced features into next-gen blockchain applications. While often compared to Subgraphs, squids are in fact far more customizable Node.js projects that can be easily configured to serve millions of dApp users at millisecond latencies through GraphQL APIs. Squids can also be used for analytics use cases, and can interface directly with popular data tools such as Grafana, Google Big Query, Amazon Athena, and even Microsoft Excel.

What makes squids different?

Of indexing solutions and API builders on the market, squids are by far the most robust in terms of speed, performance, and their growing list of features aimed at empowering developers to create sophisticated dApps.

Speeeeed

Thanks to a series of advanced engineering choices (outlined here), squids run at speeds well above other similar solutions. In a benchmark test carried out by our team, a squid replicating the Uniswap v3 Subgraph synced in 1 hour and 40 minutes. This is many times faster than the Graph’s indexing capabilities. Try it for yourself here.

With massively reduced sync times, developers are enabled to iterate faster and build better data pipelines for their applications. Rapidly evolving Web3 projects can finally stay agile as they work to achieve their business goals and respond to market demands.

Apply advanced logic

With the monolithic indexing architecture of Subgraphs and similar solutions, it just isn’t possible to efficiently carry out the kind of sophisticated data aggregations that the next generation of Web3 applications will require. Modular squids can make calls to external APIs and libraries, and interface with other data sources — including other squids!

Flexible scaling

Web3 builders require back-end infrastructure that is able to scale along with their applications. As a dApp’s user base grows, its squid can quickly be reconfigured through tweaks to the database and gateways and by setting environment variables. No other ETL allows for this kind of flexibility.

Predictable pricing

One of the biggest pain points we hear about from users of other large indexing services is frustration over the pricing model. Basically, these Web3 project teams are unable to predict indexing costs because they must pay using a volatile token! This will not be the case when retrieving and transforming data using a squid.

Currently, Subsquid (including the hosting of squids in the Aquarium) is free of charge. When we do eventually transition to a monetized service, payments will be accepted in stablecoins. We intend to offer predictable monthly subscriptions (freemium model). Your CFO will be very happy!

Join the ETH beta and migrate!

To join the beta — and receive free of charge indexing — simply fill out this form, and we’ll be in touch. We accept projects of all sizes and at all stages of development into the program.

Already have an indexer running? We want to make things as easy as possible for Ethereum developers to transition to the better solution. For this reason, we have prepared an in-depth migration guide for projects currently depending on a Subgraph.

If you require any help, we will be more than happy to provide support. You can join the SquidDevs chat on Telegram to get your questions answered by Subsquid’s core team. We also have an integrations team that may be able to complete your migration for you. You can get in touch with us directly by contacting our growth team.

As a small bonus, all teams participating in the beta will receive one Squid Squad NFT. In the future these will be very useful for Subsquid users (we’ll be talking more about this later).

If you are building on an EVM chain other than Ethereum… we’d still like to talk! We will be announcing support for other EVM L1s and L2s very soon.

About Subsquid

Subsquid is the team behind squids, a new standard for Web3 data extraction and transformation. Squids already power hundreds of applications across dozens of EVM, Substrate, and WASM-based chains. With modular architecture and a host of advanced features, Subsquid offers the most resource-efficient and developer-friendly way to build, test, and deploy customized APIs for blockchain-facing applications.