ZeroRAG: Decentralized Semantic Peer-to-Peer Routing for a Rearchitected RAG Retrieval Layer

RAG (Retrieval-Augmented Generation) has become the standard method for improving the reliability of large language model outputs: first retrieve documents relevant to the query, then place them in the prompt. But nearly all existing systems rely on centralized document storage and indexing, while in reality a large number of documents are held by organizations, devices, and individuals who, for reasons of privacy, copyright, or regulatory compliance, cannot or will not entrust them to a central party. The paper's proposed ZeroRAG attempts to solve this contradiction without introducing a coordinating authority.

The Structural Predicament of Centralized RAG

The value of RAG lies in constraining model outputs with external knowledge, but the premise of "where external knowledge is stored" is rarely questioned. Today's RAG systems habitually collect, index, and serve documents centrally, which amounts to requiring document owners to give up actual control over their data.

For personal privacy data, internal enterprise documents, or copyrighted corpora, this relinquishment is often unacceptable. Existing decentralized retrieval approaches have also failed to truly avoid this pitfall: one class of approaches reintroduces a coordinating authority and is essentially still centralized; another relies on global data structures such as DHTs, requiring network-wide continuous coordination with heavy operational costs and consistency burdens. ZeroRAG's entry point is clear—documents remain in the hands of their owners, while the retrieval layer itself is decentralized.

DHT (Distributed Hash Table) is a typical decentralized global data structure: each node is responsible for maintaining a small portion of key-value pairs according to hashing rules, and queries locate the responsible node through multi-hop routing. It is widely used in P2P systems such as BitTorrent and IPFS. However, DHTs require the entire network to reach continuous consensus on the key space partition; node joins, departures, or failures all trigger routing table updates and data migration, thus requiring network-wide coordination. For scenarios like RAG with highly variable query patterns and highly dynamic document semantics, the cost and consistency burden of maintaining a global DHT often exceed the benefits—this is precisely one of the pitfalls the paper points out that decentralized retrieval approaches cannot avoid.

How ZeroRAG Works: Lightweight State Plus Semantic Routing

ZeroRAG is a decentralized routing layer for RAG document retrieval that performs semantic query routing over an overlay network formed by peer nodes, without requiring global data structures or centralized orchestration.

Each node maintains only two types of lightweight local state: a bounded set of neighbor advertisements and a cache of recently routed queries. When a query arrives, ZeroRAG forwards it step by step using a best-first walk, continuously rewiring the overlay network in the process so that neighbor relationships evolve toward semantic relevance. This means the network self-organizes, and recall improves over time as queries accumulate, rather than relying on a pre-built global index.

The cleverness of this design lies in turning the "index" from a static asset into a dynamic process: the connections between nodes themselves carry semantic information, and every walk of a query fine-tunes the network topology. For nodes that own documents, there is no need to expose document content—only to participate in routing—so data sovereignty and retrievability can coexist.

Overlay network is a virtual network built on top of the underlying physical network, consisting of logical connections between nodes. Nodes do not need to know each other's real network locations; they only need to maintain a set of logical neighbors, and messages are forwarded along these logical edges. ZeroRAG's semantic routing runs on such an overlay network: neighbor relationships are not established randomly or by topological distance, but evolve according to the semantic similarity of the documents held by nodes, so the logical edges themselves encode semantic proximity. The advantage of an overlay network is that routing policies can be designed completely independently of underlying network details, facilitating best-first walks and dynamic rewiring.

Best-first walk borrows from the idea of heuristic search: at each step, it selects from the current node's neighbors the one that "looks closest to the query target" to forward to, rather than exhaustively trying all neighbors or performing a random walk. The basis for judging "closeness" is usually a semantic summary or vector representation carried in node advertisements. Compared with breadth-first flooding, best-first walk has a much lower message overhead; compared with random walk, it has directionality and can converge faster to the region containing relevant documents. The cost is the possibility of getting trapped in local optima, which ZeroRAG mitigates by continuously rewiring the overlay network—the network topology itself adjusts toward semantic relevance based on query history.

Experimental Performance: Recall, Overhead, and Self-Healing Capability

The paper evaluates using three real workloads and two decentralized baselines, with main conclusions concentrated in four aspects.

In terms of recall and network message overhead, ZeroRAG outperforms both baselines. More importantly, it approaches the recall and LLM answer accuracy of centralized document storage—indicating that the performance loss caused by decentralization is kept within an acceptable range.

In terms of robustness, ZeroRAG can self-heal after catastrophic node failures. Since there is no single point of dependency such as a global index, the failure of some nodes does not cause the retrieval system to collapse entirely; the network can reorganize.

In terms of scalability, experiments reached a scale of 100,000 nodes, validating the feasibility of the approach in large-scale peer-to-peer networks. For decentralized systems, increasing node scale is usually accompanied by declining routing efficiency and state bloat; ZeroRAG's bounded local state design specifically targets this pain point.

Implications for the Evolution of RAG Architecture

The significance of ZeroRAG goes beyond a retrieval optimization technique. It suggests a different RAG deployment paradigm: data does not need to move; retrieval capability can be distributed through network protocols.

This is especially attractive for regulated industries. Documents in healthcare, finance, legal, and other fields often cannot be centralized due to compliance requirements, yet LLM-assisted retrieval and question answering are genuinely needed. If the retrieval layer can work without transferring document custody, the applicability of RAG would expand significantly.

Of course, engineering implementation still has a way to go. The effectiveness of semantic routing depends on how well network scale matches query distribution; during cold start, the quality of neighbor advertisements directly affects early recall; the cost of node failure self-healing and the possibility of malicious nodes interfering with routing also need further evaluation. The paper's experiments cover three real workloads, but heterogeneity and adversarial conditions in real production environments will be more complex.

From a broader perspective, ZeroRAG represents a shift in thinking for RAG infrastructure: rather than centralizing knowledge and then retrieving it, let retrieval follow the knowledge. As data sovereignty becomes an increasingly hard constraint, this architecture of "decentralized retrieval layer, in-place data layer" may become an important complementary path alongside centralized vector databases.

Cold start in decentralized retrieval refers to the phase when the network has just been established and nodes have not yet accumulated sufficient neighbor advertisements and query caches. At this point, neighbor relationships are approximately random, semantic routing lacks reliable directional signals, and early recall is often significantly lower than at steady state. ZeroRAG's recall improvement depends on network self-organization driven by query accumulation, so cold-start quality directly determines whether the system can enter a virtuous cycle. In practice, bootstrapping mechanisms are usually needed, such as using initial semantic summaries of documents to establish seed neighbors, or mixing in random walks early on to ensure exploration coverage.

One-line Summary

ZeroRAG proposes a decentralized semantic routing layer that keeps RAG documents on owner nodes and lets retrieval be completed through network self-organization.

Paragraph Summary

The article introduces ZeroRAG, a decentralized document retrieval routing layer for RAG. Existing RAG systems generally store and index documents centrally, requiring owners to relinquish data control, which is difficult to accept in privacy, copyright, and compliance scenarios; existing decentralized approaches either reintroduce a coordinating authority or rely on global data structures such as DHTs. ZeroRAG keeps documents on owner nodes, with each node maintaining only bounded neighbor advertisements and a recent query cache, forwarding queries via best-first walk and continuously rewiring the overlay network so that topology evolves toward semantic relevance, with recall improving as queries accumulate. Experiments on three real workloads and two decentralized baselines show advantages in both recall and network message overhead, approaching the recall and LLM answer accuracy of centralized storage, self-healing after catastrophic node failures, and scaling to 100,000 nodes. The article argues that this provides a new RAG deployment paradigm for regulated industries, but engineering implementation still faces challenges such as cold start, self-healing costs, and malicious nodes.