The Evolution of Blockchain Data Indexing: From Node to Full Chain Database

1. Introduction

When discussing decentralized on-chain applications, have we ever considered the data sources these applications use? With the development of Blockchain technology, from the initial simple dApps to today's diverse financial, gaming, and social applications, the importance of data has become increasingly prominent.

In 2024, AI and Web3 have become hot topics. In the field of artificial intelligence, data is like the source of life for its growth and evolution. Just as plants need sunlight and moisture to thrive, AI systems also rely on vast amounts of data to continuously learn and think. Without data support, even the most sophisticated AI algorithms cannot exert their intended intelligence and effectiveness.

This article will delve into the development of blockchain data accessibility, focusing on a comparison between established data indexing protocols and emerging blockchain data service protocols, with particular emphasis on the similarities and differences in data services and product architecture of the emerging protocols that integrate AI technology.

2. Evolution of Data Indexing: From Blockchain Nodes to Full Chain Database

2.1 Data Source: Blockchain Node

Blockchain is often described as a decentralized ledger. Blockchain nodes serve as the foundation of the network, responsible for recording, storing, and disseminating all transaction data on the chain. Each node has a complete copy of the blockchain data, ensuring the decentralized nature of the network. However, for ordinary users, building and maintaining a node is not an easy task, as it requires specialized technology and comes with high costs.

To solve this problem, RPC node providers have emerged. They are responsible for node management and provide data services through RPC endpoints. Public RPC endpoints are free but have rate limits; private RPC endpoints offer better performance but are less efficient for complex queries. Nevertheless, the standardized API interfaces provided by node providers lower the threshold for users to access on-chain data, laying the foundation for subsequent data parsing and applications.

2.2 Data Parsing: From Prototype Data to Usable Data

The raw data provided by blockchain nodes is usually encrypted and encoded, increasing the difficulty of analysis. The data parsing process converts complex prototype data into a more understandable and operable format, which is a key link in the entire data indexing process, directly affecting the efficiency and effectiveness of blockchain data applications.

Evolution of 2.3 Data Indexers

As the volume of Blockchain data increases, the demand for data indexers is growing. Indexers organize on-chain data and send it to databases for easier querying. They provide a unified query interface that allows developers to quickly retrieve the information they need using standardized query languages.

Different types of indexers have their own advantages:

1. Full Node Indexer: Extract data directly from the full node to ensure data integrity.
2. Lightweight Indexer: Relies on full Nodes to obtain specific data, reducing storage requirements.
3. Specialized Indexer: Optimized retrieval for specific types of data or Blockchain.
4. Aggregated Indexer: Extracts data from multiple sources and provides a unified query interface.

Current mainstream indexer protocols support multi-chain indexing and customize data parsing frameworks for different application needs. The emergence of indexers has significantly improved data indexing and querying efficiency, supporting complex queries and data filtering, bringing important innovations to Blockchain data access.

2.4 Full-chain Database: Aligning to Stream Priority

As application demands become more complex, basic data indexers struggle to meet diverse query requirements. In modern data pipeline architectures, the "stream-first" approach has become a solution to the limitations of traditional batch processing, enabling real-time data processing and analysis.

Blockchain data service providers are moving towards building data streams. Traditional indexer service providers have launched real-time data stream products, such as The Graph's Substreams and Goldsky's Mirror. There are also emerging service providers like Chainbase and SubSquid offering real-time data lake services.

These services aim to address the needs for real-time parsing and comprehensive querying. By redefining on-chain data management through the lens of modern data pipelines, we can envision a future of high-performance datasets tailored for any business use case.

3. AI + Database: In-depth Comparison of The Graph, Chainbase, and Space and Time

3.1 The Graph

The Graph network provides multi-chain data indexing and querying services through decentralized nodes. Its main product models include a data query execution market and a data index caching market. The network consists of four roles: indexers, curators, delegators, and developers, working together to support the data needs of web3 applications.

The Graph has shifted to a fully decentralized subgraph hosting service, with economic incentives among participants to ensure system operation. Its core development team, Semiotic Labs, is dedicated to optimizing index pricing and user query experience using AI technology, and has developed tools such as AutoAgora, Allocation Optimizer, and AgentC, enhancing the system's intelligence and user-friendliness.

3.2 Chainbase

Chainbase, as a full-chain data network, integrates all Blockchain data onto one platform. Its features include a real-time data lake, dual-chain architecture, innovative data format standards, and an encrypted world model.

The AI model Theia of Chainbase is a key highlight. Based on NVIDIA's DORA model, Theia combines on-chain and off-chain data, deeply mining the potential value of on-chain data through causal reasoning, providing users with intelligent data services.

3.3 Space and Time

Space and Time (SxT) is committed to building a verifiable computing layer that expands zero-knowledge proof technology. Its innovative Proof of SQL technology ensures that SQL queries executed on decentralized data warehouses are tamper-proof and verifiable.

SxT collaborates with Microsoft's AI Innovation Lab to develop generative AI tools that simplify the process for users to handle blockchain data through natural language processing. In Space and Time Studio, users can experience the AI's capability to automatically convert natural language queries into SQL and execute them.

Conclusion and Outlook

Blockchain data indexing technology has evolved from the initial node data source, through the development of data parsing and indexers, to the final evolution of AI-powered full-chain data services. This process not only improves the efficiency and accuracy of data access but also brings users an intelligent experience.

With the continuous development of new technologies such as AI and zero-knowledge proofs, blockchain data services will become further intelligent and secure. In the future, blockchain data services will continue to play an important role as infrastructure, driving innovation and progress in the industry.