New Block Model to Accelerate Chain Integration
Developed for The Graph, Substreams is a powerful indexing technology that allows you to apply transformations to blockchain data. Substreams provides a rich data model to consume; however, providing such a complete data model is not an easy task, and often requires work from the core developer team of the blockchain at hand, or deep instrumentation of nodes.
To accelerate the integration of new supported chains, StreamingFast has introduced a base data model (where DetailLevel == BASE) as opposed to our native Extended Ethereum data model. This base data model makes it easier to roll out new chains and is still useful for most use cases.
The Extraction Layer
The extraction layer of Substreams is Firehose, a technology developed by StreamingFast to extract data from blockchain nodes in a highly efficient manner.
The Firehose engine extracts every block of the blockchain and makes it available to consumers through a Protobuf schema.
The Block Protobuf schema depends on the data model of the blockchain: all EVM-compatible chains share the same Block model; however, chains with a different data model, such as Solana or NEAR use a different Block Protobuf model.
Developing a full Firehose integration for a new chain is not an easy task; it requires a deep technical understanding of the Firehose technology and takes a considerable amount of time to develop it. It also requires ongoing support as chains update their node software. With the desire to support as many chains as possible, upgrade support alone can become a monumental task
Blocks with Base Data
In order to facilitate the integration of new chains in the Ethereum ecosystem with Firehose and Substreams, the StreamingFast team has introduced different levels of details in Blocks. In Firehose integrations using Base Blocks, some of the fields within a Block will return empty.
Extracting Base Blocks requires running only a basic full node for real-time segments and an archive node to process historical data. However, extracting Extended Blocks requires a Firehose-enabled geth node (or other instrumentation-based implementation), thus making the process more complex.
The following table defines which pieces of data are not available on Base Blocks.
Identify Base Blocks
The Protobuf used for Base Blocks is the same as the one used for Extended Blocks. Therefore, the Protobuf will have missing fields when using Base Blocks.
For example, if you try to access the balance changes of a Base Block, you will get an empty list.
You can verify if the Block is Base or Extended by using the detail_level field in the Protobuf.
Where are Base Blocks and Extended Blocks Supported?
Currently, most chains use Extended Blocks. However, upcoming supported chains will most likely use Base Blocks.
The following diagram illustrates what EVM-compatible chains use Base Blocks.
At StreamingFast, we are always listening to our community. Let us know on Discord if there is a specific chain that you would like to see supporting Base Blocks!
