Netflix-Grade Resume Playback: Replacing Redis with Dragonfly DB in Practice

"Continue Watching" may look like the most unremarkable feature in a video product, yet it can be the dividing line in a system design interview between an average candidate and a senior engineer. A YouTube tech blogger spent an entire video breaking down how to build a resume-playback system from scratch that can withstand Netflix-scale traffic, using Dragonfly DB, Postgres, and the ImageKit player. This article distills the architectural trade-offs and engineering details, and adds analysis alongside the code implementation.

One Feature, Three Questions That Must Be Answered

The core logic of Continue Watching boils down to three questions: Where did the user stop? What is the watch order of each video? Is the user currently watching? These three questions map to video progress, the recently watched list, and real-time online status, respectively.

According to the data cited by the blogger, Netflix has roughly 325 million paid subscribers (he admits this figure may be outdated and estimates it at around the 300 million mark). Even assuming only 10 million users are watching simultaneously—and many of them watch on both phone and TV—with each player sending a heartbeat request every 5 seconds, the write pressure would instantly overwhelm a single database.

This is the starting point of the problem: heartbeats cannot be skipped, but they absolutely must not be written directly to a relational database.

Why Postgres Can't Handle Heartbeat Writes

A single Postgres instance can handle roughly 10,000 to 50,000 writes per second, whereas heartbeats every 5 seconds from 10 million users mean millions of write requests per second. The gap cannot be closed by adding indexes or tuning parameters.

The deeper reason lies in Postgres's design philosophy: it is built as a system that never loses data. Every update goes through WAL (write-ahead logging) to disk, acquires row locks, and old rows are not deleted immediately but left for the vacuum process to clean up. When heartbeats from the same account arrive simultaneously from TV, phone, and laptop, row-lock contention further slows writes. These mechanisms guarantee reliability, but in ultra-high-frequency, short-lived write scenarios they become a burden.

The blogger specifically notes that this analysis itself is a common system design interview topic—what the interviewer wants to hear is precisely "why can't we just write directly to the database."

WAL (Write-Ahead Logging) is the core mechanism relational databases use to guarantee durability and crash consistency. Before any data modification is written to the actual data page, it must first be sequentially appended to the WAL log and flushed to disk (fsync). This way, even if the process crashes or power is lost, unflushed transactions can be recovered by replaying the log after restart. The cost is at least one extra disk I/O and fsync call per write, and fsync latency is typically in the millisecond range. For high-frequency small writes every 5 seconds at millions per second, the WAL flush overhead is multiplied, becoming a throughput bottleneck. In contrast, in-memory databases like Dragonfly do not need to synchronously flush to disk on every write, and the write path is extremely short—this is the fundamental reason the write-behind pattern can offload pressure from Postgres.

Core Architecture: Separating Hot Data from Durable Data

The key insight of the solution is to separate hot data from durable data.

• Hot data: the heartbeat position every 5 seconds—changes extremely fast, read extremely frequently, but only the latest value matters and no history needs to be retained.
• Durable data: the position a user wants to resume from after returning from a trip two days later—must never be lost.

Hot data is written to Dragonfly DB (an in-memory database), with read latency below milliseconds and overwrites costing almost nothing. The Continue Watching card on the homepage reads directly from here. A flush worker batch-writes the latest positions to Postgres every ~60 seconds; this interval was chosen arbitrarily by the blogger and can be configured to two or five minutes.

This pattern has a formal name in system design: write-behind caching. Data is written to the cache first, then batch-persisted to the database according to a time window, rather than writing to the database first and then updating the cache. The combination of 10 million users, 5-second heartbeats, and a 60-second persistence window means database write volume is directly reduced by 12x.

Fault-Tolerance Trade-off: Losing One Minute Is Acceptable

What if Dragonfly DB restarts and memory is wiped? The blogger's answer is candid: almost no impact. At most, about 60 seconds of data not yet persisted is lost. If a user opens the homepage and finds the resume record missing, the system can pull the user's unfinished videos from Postgres and repopulate Dragonfly.

This attitude of "clearly knowing what will be lost and how to recover it" is more aligned with the reality of high-concurrency systems than pursuing absolute zero data loss.

Why Dragonfly Instead of Redis

The comparison data cited by the blogger is that Dragonfly's QPS reaches 25x that of Redis. The difference comes from architecture—Dragonfly uses a shared-nothing multithreaded architecture that can utilize every CPU core on the machine, whereas Redis's single-threaded model can only be solved by stacking clusters at scale, with very substantial bills.

The most critical point for developers is: Dragonfly is a drop-in replacement for the Redis protocol. The code uses the ordinary ioredis client; only a connection URL needs to change, and the rest of the code stays untouched.

Dragonfly also supports periodic snapshot persistence, so in-memory data can be automatically recovered after a restart, alleviating the inherent impression that "Redis is purely in-memory and everything is gone after a power loss." The blogger demonstrated a local startup configuration via Docker Compose—note that each thread needs 256MB of memory reserved, and the thread count is configured by multiplying as needed.

Redis's single-threaded model refers to its command execution engine processing all client requests serially on a single main thread. The benefit is natural avoidance of lock contention and simple, consistent semantics. But single-threaded means a single instance can only utilize one CPU core; when QPS approaches the single-core ceiling, the only option is horizontal scaling by sharding into multiple Redis instances (a cluster), which drives up operational complexity and cloud bills. Dragonfly's shared-nothing multithreaded architecture instead partitions the key space by hash across multiple threads, with each thread independently handling the keys it owns. Threads share almost no state and take no global locks, so they can fully saturate multi-core CPUs. Note that multithreaded design imposes additional constraints on the semantics of cross-key operations (such as transactions and multi-key atomic commands), which is important background for understanding the co-slot requirements of hash tags and pipelines discussed later.

Three Data Structures Power the Entire Feature

In Dragonfly, each user's three types of data are stored in different structures:

• Hash: stores watch progress state (position in seconds and other fields)
• Sorted Set: a recently watched list sorted by time; the worker uses ZREVRANGE to retrieve the latest N entries
• String + TTL: marks the "currently watching" state, with an expiration of about 30 seconds

These three operations are submitted together via pipeline rather than three separate network round trips. The blogger emphasizes that in-memory operations are already faster than disk; if you then write them one by one and wait for each acknowledgment, you are wasting the advantages of an in-memory database. One network round trip with millisecond-level return is the basic requirement of this design.

The Postgres schema is lean: the videos table stores ID, title, URL, and duration; the watch_progress table stores user ID, video ID, position in seconds, last watched time, and a completion flag. No authentication system was introduced; user switching is simulated.

Pipeline is a batch submission mechanism at the Redis protocol level: the client writes multiple commands to a TCP connection at once, the server executes them in order and returns the results packaged together, thereby compressing multiple "send—wait for acknowledgment" network round trips into one. It is not a transaction; it does not guarantee atomicity, nor does it guarantee that other clients' commands won't be interleaved between commands, but it significantly reduces RTT (round-trip time) overhead. In Dragonfly, the benefit of pipelines is even more pronounced because the multithreaded engine can schedule work for different slots in parallel within a single batch. For the scenario in this article, merging the three write operations—progress Hash, ordered Sorted Set, and online TTL—through a pipeline is the key step that truly translates the millisecond-level in-memory advantage into end-to-end latency.

Hash Slots: The Detail Interviewers Love to Probe

Redis Cluster uses a fixed 16,384 hash slots—a magic number from the Redis team's early design decisions, still widely used in production today. The modulo is fixed at 16,384 rather than the current number of servers, so slot mapping remains unchanged during scaling up or down.

But ordinary hashing has a trap: the three keys for the same user (progress, order, online status) may land on different nodes after hashing, while pipeline batch operations require them to execute on the same node. The solution is hash tags—wrap the part of the key that needs to share a slot in curly braces, e.g. watch:{user1}. The hash function only computes the content inside the braces, so all keys for the same user necessarily land in the same slot.

The hash function used by the blogger is CRC16, not the SHA family—the latter is designed for attack resistance, consumes more CPU, and is unnecessary for slot allocation.

Five Reusable System Design Principles

The blogger concludes with five lessons worth listing separately:

1. Hot/cold separation: not every write deserves to be persisted; if a write can be deferred, defer it.
2. Batch writes: pull from the cache in batches and persist in batches, rather than writing one by one.
3. Choose the right data structures: Hash for state, Sorted Set for ordering, TTL for timeliness.
4. Plan for cache loss: clearly define the scope of loss and the backfill path so nothing unexpected happens in production.
5. Shard by user: use hash tags to ensure a user's keys do not drift across nodes.

Hidden Engineering Considerations in Player Selection

The ImageKit player used in the demo is open-source and free. What the blogger values is its adaptive bitrate streaming (HLS/DASH) capability—video is delivered in chunks and automatically switches between resolutions like 360p based on network conditions, avoiding loading an entire large file at once. It also provides subtitles, chapters, seek preview thumbnails, and playback event callbacks. After uploading an MP4, transcoding and HLS distribution are handled automatically by the platform.

One pitfall worth mentioning: in React Strict Mode, the player component is destroyed once on first mount, causing the video element to be removed and playback to crash. The blogger disabled Strict Mode in the code with a comment explaining why, and said he would file an issue with the repository. Production builds do not enable Strict Mode, so there is no impact online.

Code Organization and AI-Assisted Development

The project uses a monorepo: the api app contains heartbeat reception, Dragonfly read/write, and the flush worker; the web app handles the player and homepage cards. The worker logic periodically scans keys, retrieves them in batches via the Redis pipeline, and writes to Postgres in a single database transaction.

The blogger's attitude toward AI coding is pragmatic: once the concepts are clearly understood, the code can be generated by agents like Claude and Codex. What developers need to master is architectural judgment and problem-locating ability—for example, why it crashes under Strict Mode, and why keys need curly braces. He also mentions that the repository provides a "copy to LLM" Markdown entry point to facilitate end-to-end implementation by agents.

It is worth emphasizing that the blogger candidly admits he implemented a similar feature by hand years ago, and the first version wrote all heartbeats directly to MySQL; the database was crushed within two days. This lesson precisely confirms the core argument of this article: the difficulty of Continue Watching has never been the feature itself, but every write decision at scale.

One-line Summary

Use Dragonfly to handle hot heartbeat data and Postgres for durability, building a resume-playback system that can withstand Netflix-scale traffic.

Paragraph Summary

Using the "Continue Watching" feature as a lens, the article breaks down the architectural trade-offs of a high-concurrency resume-playback system. The core contradiction: heartbeats every 5 seconds generate millions of writes per second, far exceeding a single Postgres instance's capacity of 10,000 to 50,000 writes per second, while WAL flushing and row-lock contention further amplify the bottleneck. The solution is hot/cold separation—Dragonfly DB handles high-frequency heartbeat hot data, and through a write-behind caching pattern, a flush worker batch-persists to Postgres every ~60 seconds, reducing database write volume by about 12x at the cost of losing at most one minute of data, which can be backfilled from Postgres. The article also details how Hash, Sorted Set, and String+TTL structures work with pipeline submission, and how hash tags ensure the same user's keys share a slot, concluding with five reusable system design principles.