RPC Poller — Bringing Substreams and Firehose to new chains 10x faster!

StreamingFast introduces the RPC Poller, a novel extraction technology to bring Firehose and Substreams to new chains 10x faster.

Firehose and Substreams work together to retrieve blockchain data faster than other indexing technologies. At the same time, the rich Substreams data model allows customers to build complex and reliable applications. However, this all comes at the cost of developing a complex and time-consuming integration for every chain.

Within just the EVM ecosystem, there are hundreds of blockchains serving different purposes, with more yet to come. Building a Substreams integration for each EVM blockchain requires a lot of work and time, thus slowing down the adoption of Substreams. Having full Substreams support for a chain involves running a Firehose-enabled node, which traces the execution of every transaction; this is how Substreams is able to provide such a rich data model.

The RPC Poller does not rely on a Firehose-enabled node; instead, it retrieves the data from the existing standard JSON-RPC endpoints of the chain. The RPC endpoints are defined by the Ethereum Execution API Specification, therefore, every EVM chain exposes the same API interface. This means that extracting data from Optimism or Arbitrum (or any other EVM chain) using the RPC endpoints requires no extra tooling, thus facilitating the integration of new chains 10 times faster!

However, some parts of the Substreams data model can’t be extracted using the RPC endpoints (for example, Calls or Balance Changes). Substreams using RPC-powered chains will only have a basic data model, which is still useful for most use cases. While the data is not as rich as what would be provided by a Firehose instrumented node, it is at parity with the data of other RPC providers, but with streaming capabilities!

You can read more about the reduced data model of Substreams in this article.

The development of the StreamingFast RPC Poller is a huge improvement for the adoption of Substreams, as many more chains will be available throughout 2024! Is there any particular chain you would like to see supported for Substreams soon? Let us know on Discord, maybe the team is already working on it!