Delta: A Deep Dive into a Highly Available, Strongly Consistent Storage System Built on Chain Replication

What Is the Delta Storage System

Delta is a distributed storage system designed for high availability and strong consistency, with Chain Replication as its core architectural pattern. In the distributed storage space, balancing strong data consistency with high availability has always been one of the trickiest engineering trade-offs. Delta attempts to find a better equilibrium between the two through this classic yet elegant approach.

It's worth noting that this article is based on a 2022 technical document shared on Hacker News that received limited traction (7 points, no comments at the time). As such, this post focuses primarily on providing background context around chain replication as the central technology, helping readers understand the design philosophy behind systems like Delta.

The Core Principles of Chain Replication

Chain replication was first introduced by van Renesse and Schneider in their seminal paper. Rather than using a traditional primary-backup topology, it organizes replica nodes into an ordered chain, with each end of the chain serving a distinct role.

Write Path: Head to Tail

All write requests enter at the head node of the chain and propagate downstream sequentially. Each node applies the update locally before forwarding it to the next node. A write is only considered committed — and a success response returned to the client — once the update has been acknowledged by the tail node.

The key advantage of this design is that data durability and consistency are naturally guaranteed by the chain's propagation order. When the tail acknowledges an update, it means every node in the chain already holds that update.

Read Path: All Reads Served by the Tail

In contrast to writes, all read requests are handled exclusively by the tail node. Since the tail holds the latest committed state that has been confirmed by the entire chain, reads from the tail provide strong consistency (linearizability) guarantees directly — without requiring multi-replica coordination at read time, as Quorum-based protocols do.

Linearizability is the strongest consistency model in distributed systems. It requires that every operation appear to take effect atomically at some point between its invocation and completion, with all operations forming a global order consistent with real-time ordering. In plain terms: once a write returns successfully, any subsequent read by any client must reflect that write.

Quorum-based protocols (e.g., Raft's ReadIndex, Paxos read quorum) must contact a majority of nodes during reads to satisfy linearizability, incurring additional network round-trip overhead. Chain replication's elegance lies in the fact that the tail node's local state is inherently the "globally committed latest state" — since a write confirmed by the tail means all nodes in the chain already hold the data — so no cross-node coordination is needed for reads. This significantly reduces the complexity of the read path in practice.

Why Choose Chain Replication

Compared to consensus protocols like Paxos and Raft that rely on majority quorums, chain replication offers clear engineering appeal in certain scenarios:

• Simple, concentrated read throughput: Read operations only need to reach the tail node — the logic is straightforward and latency is predictable.
• Distributed write load: Each node only communicates with its immediate neighbor rather than broadcasting to all replicas, reducing per-node network pressure.
• Strong consistency by construction: Consistency is guaranteed by the chain topology itself, not by complex voting mechanisms.

That said, chain replication is not without costs. Write latency scales linearly with chain length — the longer the chain, the more nodes a write must traverse, and the higher the end-to-end latency. This is exactly why systems like Delta typically require careful tuning of the trade-off between replica count and write performance.

Both Paxos and Raft implement consensus through a majority quorum mechanism: in a cluster of 2f+1 nodes, an operation can be committed once a majority (f+1 nodes) confirms it, tolerating up to f simultaneous node failures. During writes, the leader broadcasts log entries to all followers in parallel and waits for majority acknowledgment, placing a heavy fan-out network burden on the leader node. Raft is generally considered easier to understand and implement than Paxos, and has a mature open-source ecosystem (etcd, TiKV, etc.) — which is part of the context behind the observation at the end of this article that "chain replication has a relatively niche engineering and operational knowledge base."

The fundamental difference between the two approaches: Quorum protocols use "majority overlap" to ensure information carries over across decision rounds, while chain replication uses strict sequential propagation to replace voting — essentially trading latency for topological simplicity.

How High Availability Is Achieved

Plain chain replication has an obvious weakness: any single node failure breaks the entire chain. For Delta to achieve "high availability," it must layer failure detection and chain reconstruction on top of chain replication.

The typical approach is to introduce a dedicated control plane (common in improvements like CRAQ, often backed by a component such as ZooKeeper) responsible for monitoring the health of nodes in the chain. When a node fails:

• If the head node fails, its downstream neighbor takes over as the new head;
• If the tail node fails, its upstream neighbor is promoted to the new tail;
• If a middle node fails, its predecessor and successor are directly connected, bypassing the failed node.

Through this dynamic reconstruction, the system can quickly resume service when some nodes fail — simultaneously delivering strong consistency and high availability. This is precisely the dual goal captured in Delta's stated mission: "Highly available, strongly consistent."

CRAQ (Chain Replication with Apportioned Queries) is an important improvement on classic chain replication, proposed by Terrace and Freedman in 2009. It allows all nodes in the chain — not just the tail — to serve read requests, horizontally scaling read throughput across the entire chain. This is achieved by maintaining both a "committed version" and a "dirty version" for each object: if a non-tail node's latest state for an object is committed (i.e., no update is currently propagating), it can respond to reads locally; otherwise, it must query the tail for the latest committed version number.

Strongly consistent coordination services like ZooKeeper are commonly used as the control plane for chain replication systems, responsible for maintaining the chain's membership view, coordinating node joins and departures, and triggering chain reconstruction on failure — cleanly separating data plane and control plane responsibilities.

Use Cases and Limitations

Storage systems like Delta, built on chain replication, are best suited for read-heavy, write-light workloads with strict consistency requirements — for example, metadata storage, configuration management, and distributed lock services. In these scenarios, concentrating reads at the tail node can deliver excellent throughput, while the write latency growth with chain length remains relatively manageable.

Its limitations are equally clear:

• Write latency is sensitive to chain length, making it unsuitable for very large replica sets;
• Chain reconstruction during failures can cause brief availability disruptions;
• Compared to the mature Raft ecosystem, chain replication has a relatively niche engineering and operational knowledge base.

Conclusion

Delta illustrates the enduring viability of chain replication as a replication and consistency mechanism. On the classic distributed systems question of "strong consistency vs. high availability," it takes a different path from mainstream Quorum protocols — using an ordered chain topology to achieve simpler consistency guarantees and a more focused read path. For engineering teams evaluating distributed storage solutions, understanding the trade-offs inherent in chain replication can help inform better technology choices for specific business contexts.

(Note: Due to the limited information in the original source material, portions of this article describing Delta's specific implementation details are extrapolated from general chain replication principles. For actual system behavior, please refer to the official technical documentation.)