The Evolution of Blockchain Data Indexing: From Nodes to Full-Chain AI Database

1. Introduction

Since the first batch of decentralized applications ( dApp ) emerged in 2017, the Blockchain ecosystem has flourished, giving rise to numerous financial, gaming, and social dApps based on different Blockchains. However, the various data sources that these applications rely on during the interaction process merit our in-depth consideration.

In 2024, artificial intelligence and Web3 have become hot topics. In the field of AI, data is like the source of life, driving the learning and evolution of systems. Without the support of massive amounts of data, even the most sophisticated AI algorithms struggle to realize their potential.

This article will delve into the evolution of data indexing in the context of blockchain data accessibility during the industry's development process. We will also compare the traditional data indexing protocol The Graph with the emerging blockchain data service protocols Chainbase and Space and Time, exploring the similarities and differences of these new protocols that integrate AI technology in terms of data services and product architecture.

2. The Evolution of Data Indexing: From Blockchain Nodes to Full-Chain Database

2.1 Data Source: Blockchain Node

Blockchain is known as a decentralized ledger, and nodes are the foundation of this network. Each node maintains a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node not only has a high technical threshold but also requires bearing expensive hardware and bandwidth costs.

To solve this problem, RPC Node providers have emerged. They are responsible for the operation and maintenance of nodes, providing data access services to users through RPC endpoints. Although public RPC endpoints are free, they have rate limits that may affect the user experience of dApps. Private RPC endpoints, while offering better performance, still do not have high efficiency for complex queries and are difficult to scale across networks. Nevertheless, the standardized API interfaces of node providers have lowered the threshold for users to access on-chain data, laying the foundation for subsequent data parsing and application.

2.2 Data Parsing: From Raw Data to Usable Data

The raw data provided by blockchain nodes is often encrypted and encoded, which ensures the integrity and security of the data but also increases the difficulty of parsing. For ordinary users and developers, directly handling this data requires a large amount of specialized knowledge and computing resources.

Therefore, the data parsing process has become particularly important. By converting complex raw data into a format that is easy to understand and operate, users can utilize this data more intuitively. The quality of data parsing directly affects the efficiency and effectiveness of blockchain data applications, and is a key link in the entire data indexing process.

Evolution of Data Indexers 2.3

As the volume of Blockchain data surges, the demand for data indexers is becoming increasingly urgent. Indexers are responsible for organizing on-chain data and storing it in databases for querying. They index Blockchain data and provide SQL-like query languages such as GraphQL interfaces, making data readily available. Indexers offer developers a unified query interface, greatly simplifying the data retrieval process.

Different types of indexers have their own advantages:

1. Full Node Indexer: Directly extract data from full nodes to ensure data completeness and accuracy, but requires a large amount of storage and processing power.
2. Lightweight Indexer: Relies on full nodes to obtain specific data on demand, reducing storage requirements but potentially increasing query time.
3. Dedicated Indexer: Optimized for specific types of data or Blockchain, such as NFT data or DeFi transactions.
4. Aggregate Indexer: Extracts data from multiple Blockchains and sources, including off-chain information, providing a unified query interface, suitable for multi-chain dApps.

Currently, the storage requirements for Ethereum archive nodes vary between clients, ranging from 3TB to 13.5TB. In the face of such a massive amount of data, mainstream indexing protocols not only support multi-chain indexing but also tailor data parsing frameworks to different application needs, such as The Graph's "Subgraph" ( Subgraph ) framework.

The emergence of indexers has significantly improved data indexing and query efficiency. Compared to traditional RPC endpoints, indexers can efficiently handle large amounts of data, supporting complex queries and data filtering. Some indexers also support aggregating data sources from multiple Blockchains, avoiding the need for multi-chain dApps to deploy multiple APIs. By operating in a distributed manner, indexers not only provide stronger security and performance but also reduce the interruption risks that centralized RPC providers may pose.

( 2.4 Full-Chain Database: Transitioning to Stream-First Mode

As application demands become increasingly complex, basic data indexers and their standardized index formats are gradually struggling to meet diverse query needs, such as cross-chain access or off-chain data mapping.

In modern data pipeline architecture, the "stream-first" approach has become a solution to the limitations of traditional batch processing, enabling real-time data processing and analysis. Blockchain data service providers are also moving towards building data streams, such as The Graph's Substreams, Goldsky's Mirror, and the real-time data lakes provided by Chainbase and SubSquid.

These services aim to address the demand for real-time parsing and comprehensive query capabilities of Blockchain transactions. By redefining on-chain data challenges from the perspective of modern data pipelines, we can view the potential of data management, storage, and provision from a fresh angle. By considering the indexer as a data stream rather than a final output, we can envision the possibility of tailoring high-performance datasets for any business use case.

3. AI + Database: In-depth Comparison of The Graph, Chainbase, and Space and Time

) 3.1 The Graph

The Graph network provides multi-chain data indexing and querying services through a decentralized network of nodes. Its core products include a data query execution market and a data indexing cache market, serving the product query needs of users.

Subgraphs ### are the fundamental data structures of The Graph network, defining how to extract and transform data from the Blockchain into a queryable format. The network consists of four key roles: indexers, curators, delegators, and developers, who work together to provide data support for web3 applications.

The Graph has fully transitioned to a decentralized subgraph hosting service, with participants ensuring the system's operation through economic incentives. Recently, The Graph's core development team, Semiotic Labs, has utilized AI technology to optimize index pricing and user query experience, developing tools such as AutoAgora, Allocation Optimizer, and AgentC, further enhancing the system's intelligence and user-friendliness.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track]###https://img-cdn.gateio.im/webp-social/moments-cf9a002b9b094fbbe3be7f611001b5c1.webp(

) 3.2 Chainbase

Chainbase is a full-chain data network that integrates all Blockchain data into one platform. Its featured functionalities include:

• Real-time Data Lake: Provides a real-time data lake specifically for blockchain data streams.
• Dual-chain architecture: Built on Eigenlayer AVS for the execution layer, forming a parallel dual-chain architecture with the CometBFT consensus algorithm.
• Innovative Data Format Standard: Introduced "manuscripts" data format standard.
• Cryptographic World Model: Combining AI model technology to create an AI model Theia that can understand and predict Blockchain transactions.

Chainbase's AI model Theia is based on NVIDIA's DORA model, combining on-chain and off-chain data analysis encryption patterns, responding through causal reasoning to provide users with intelligent data services.

3.3 Space and Time

Space and Time (SxT) is dedicated to building a verifiable computation layer that extends zero-knowledge proofs on a decentralized data warehouse. Its core technology, Proof of SQL, is an innovative zero-knowledge proof technique that ensures SQL queries executed on the decentralized data warehouse are tamper-proof and verifiable.

SxT collaborates with Microsoft's AI Joint Innovation Lab to develop generative AI tools that allow users to process blockchain data through natural language. In Space and Time Studio, users can experience inputting natural language queries, with AI automatically converting them to SQL and executing the queries.

![Reading, Indexing to Analysis, Brief Overview of the Web3 Data Indexing Track]###https://img-cdn.gateio.im/webp-social/moments-97443cbd177ac4ffd1665da670ffbf12.webp(

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data source, through the development of data parsing and indexers, to the fully empowered AI chain data services, undergoing a gradual improvement process. The continuous evolution of these technologies has not only improved the efficiency and accuracy of data access but also brought users an unprecedented intelligent experience.

Looking ahead, with the continuous development of new technologies such as AI and zero-knowledge proofs, Blockchain data services will become further intelligent and secure. As an infrastructure, Blockchain data services will continue to play an important role in industry advancement and innovation.

![Reading, indexing to analysis, a brief overview of the Web3 data indexing track])https://img-cdn.gateio.im/webp-social/moments-0742180b7da8a9dcddafc465a4dba9cb.webp(