Subsquid’s biggest release yet… real-time indexing!

Back in June, we made what had been the largest update to the Subsquid indexing stack since our project’s founding. Highlights of the FireSquid release included massive sync speed increases (up to and beyond 50k blocks per second), lower maintenance Archives, and updated GraphQL interfaces, including a human-readable, SQL-like, experience.

In the months following, our development team pushed several other major updates to production. In October, we expanded beyond Substrate, launching full Ethereum support, and have since deployed Archives on around 20 other EVM chains and ETH L2s.

Subsquid releases Q3 2022 - Q3 2023

June 2022: FireSquid release (Web3's fastest ever indexer);
July 2022: Improved GraphQL interfaces;
August 2022: Full ink! smart contract support;
October 2022: Ethereum indexing;
December 2023: Sidechains (i.e. Avalanche, Polygon, BNB chain);
January 2023: File stores (export datasets to external DBs, CSV, Parquet, JSON, etc);
February 2023: ETH L2s (i.e. Arbitrum); 
March 2023: Giant Squid release, CLI update (direct Postgres access, interactive UI), more L2s;
April 2023: ArrowSquid release (Real-time indexing!), more L2s;

Throughout Q1, our major releases have been focussed around increasing the flexibility of squid development. This has included improvements to the store interface, enabling developers to ‘bring their own database,’ and to export datasets to the target of their choice (i.e. CSV, Apache Parquet, and JSON sores, or in S3 or BigQuery). A massive update to Squid CLI added an interactive UI that appears right in the terminal, which gives developers direct access to the Postgres databases of squids deployed in the Aquarium, Subsquid’s managed service.

We are excited to announce that we are currently putting the finishing touches to the next generation of our SDK and Archives. This release will be called ArrowSquid (yes, like FireSquid this is also a real species, check it out on Wikipedia). The ArrowSquid is currently being tested by our QA team, and will be released in a public beta in the next few weeks.

Here’s what you can expect from the ArrowSquid release:

⚡ Real-time data indexing: Squid SDK will offer full support for indexing unfinalized blocks, reducing query latencies down to ‘real-time.’ The SDK will handle reorgs out-of-the-box.

🏹 Historical data, supercharged: We have implemented Apache Arrow for data ingestion from Archives. This will lead to a considerable boost in performance for ‘heavy’ squids.

☕ Simplified interfaces: Rapidly filter through logs and transactions to mine any volume of on-chain data at scale.

🔎 Trace data: It will be possible to create squids that track internal transactions (to be released shortly after ArrowSquid’s initial launch).

TLDR

The purpose of this article was primarily to give a ‘head’s up’ to our current users, who may wish to upgrade to ArrowSquid right away, following its release. The current SDK and Archives will remain available and supported until their end-of-life notice. Upon the official launch of ArrowSquid, we will also release a thorough migration guide for developers. Our integrations and DevRel teams will be on hand to assist throughout this process.

For further updates, make sure to follow us on Twitter, join our Discord, and become a part of the SquidDevs Telegram group.