Substreams reach General Availability
Today, one year after the very first line of code being written, we’re happy to announce that the StreamingFast Substreams engine has reached General Availability. It is deemed ready for production and will now be officially supported, both the software and the service hosted by StreamingFast. Supported networks will start with Ethereum and Polygon.
Substreams is a powerful blockchain indexing technology, developed for The Graph Network.
Substreams enables developers to write Rust modules, composing data streams alongside the community, and provides extremely high-performance indexing by virtue of parallelization, in a streaming-first fashion
Substreams have all the benefits of StreamingFast Firehose, like low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.
StreamingFast provides free endpoints to Substreams until the technology is made available on The Graph’s Decentralized Network.
Substreams have matured since its unveiling last June, and have acquired powerful parallelization capabilities, turning the system into a full-blown distributed processing system, akin to BigQuery — where, instead of SQL, you get to pass in WASM code to transform and aggregate data over the chain’s history, as well as live incoming blocks.
This release comes with a new Terminal-based GUI for inspecting the output of modules, as well as new parameterizable module inputs. Another recent release brought a “production mode”, which is a high speed parallel forward-processing mode designed for production, once your development is done. By contrast, development mode brings in detailed logs, all modules’ outputs, including store outputs and store snapshots, for inspection. The Rust crates also have matured and multiplied for a bunch of different use cases.
Most notably, the number of data sinks (or data loaders) have increased, thanks for the efforts of multiple teams:
- Pinax Network released: Prometheus, Winston, CSV file and a Google Sheets sink!
- Messari is about to release a Parquet files sink, and together with the other files options made available now provides ingestion into BigQuery, Clickhouse, Redshift and other large scale MPP engines
- Edge & Node is about to release: graph-node integration for Substreams-based Subgraphs
- We, StreamingFast, released: Postgres, Key/value store, MongoDB, CSV & JSON Files
Many thanks to those who have contributed to the ecosystem of tools, provided feedback or beta-tested the software this year:
- Pinax Networks, along with their contribution of multiple sinks, being the first to host the Substreams runtime, and their ownership and maintenance of a full blown Firehose integration for Antelope.
- Messari for investing time in Substreams development and improvement of the developer experience as well as sharing their thoughts on proper data design,
- Edge & Node, GraphOps and Sebastian Lorenz, who jumped in and started developing Substreams as well as providing feedback on the experience.
Also, this year has seen it’s largest growth of protocol integrations that are now Firehose-enabled (and thus are Substreams-ready), by different teams:
- Polygon, BNB, NEAR and Solana (by StreamingFast),
- Cosmos Hub, Osmosis and Juno (by Figment),
- Arweave (by ChainSafe),
- Antelope (by Pinax).
- Aptos (by Aptos)
- StarkNet (by zkLend)
and work is completed (by ChainSafe) to enable the Erigon client to speak Firehose.
We’re excited by the prospects this technology brings to developers all around, and by future opportunities this will bring.
We would love to hear your feedback and answer any questions you may have, so why not join our Discord server and chat with us directly?
— — —
About StreamingFast
StreamingFast is a web3 builder and investor. As a core developer on The Graph, it excels at building massively scalable open-source software for processing and indexing blockchain data. Founded by a team of serial tech entrepreneurs, the company has deep expertise in large-scale data science. Its core innovations, the Firehose and Substreams, are files-based and streaming-first approach that enables high-performance indexing on high throughput chains.
You can follow StreamingFast on Twitter and on Discord.
About The Graph
The Graph is the indexing and query layer of web3. Developers build and publish open APIs, called subgraphs, that applications can query using GraphQL. The Graph currently supports indexing data from over 39 different networks including Ethereum, NEAR, Arbitrum, Optimism, Polygon, Avalanche, Celo, Fantom, Moonbeam, IPFS, Cosmos Hub and PoA with more networks coming soon. To date, 74,000+ subgraphs have been deployed on the hosted service. Tens of thousands of developers use The Graph for applications such as Uniswap, Synthetix, KnownOrigin, Art Blocks, Gnosis, Balancer, Livepeer, DAOstack, Audius, Decentraland, and many others.
The Graph Foundation oversees The Graph Network. The Graph Foundation is stewarded by the Technical Council. Edge & Node, StreamingFast, Semiotic Labs, The Guild, Messari and GraphOps are seven of the many organizations within The Graph ecosystem.
Join The Graph community by introducing yourself in The Graph Discord for technical discussions, join The Graph’s Telegram chat, and follow The Graph on Twitter, LinkedIn, Instagram, Facebook, Reddit, and Medium! The Graph’s developers and members of the community are always eager to chat with you, and The Graph ecosystem has a growing community of developers who support each other.
