DeltaStreamer: How Human-Like Agents Understand Streaming Video

Streaming video understanding has long faced an awkward reality: systems must decide what to remember and what to forget without knowing what questions will be asked in the future. Compress too aggressively and you lose critical details; compress too loosely and you can't run in real time. Worse still, once details are discarded, they're nearly impossible to recover.

The newly proposed DeltaStreamer attempts to break out of this dilemma. Instead of compressing video history into a single summary, it mimics the human approach of "noting changes as you watch, then going back to verify when asked," combining continuous observation with active investigation.

Two Fundamental Flaws in Existing Methods

The paper identifies two interrelated problems with current streaming video understanding approaches.

First, event transitions get blurred. To control memory usage, many methods compress consecutive frames into dense descriptions, but in doing so, "what changed" often gets flattened out, leaving only "what's in the frame." For queries that require understanding the sequence of actions or state transitions, this representation is inherently disadvantaged.

Second, missing visual evidence is unrecoverable. Compression is lossy—once a detail is deemed unimportant during the observation phase and not retained, the system has no mechanism to retrieve it when a later query needs it. This differs from humans—a person might not remember a license plate number, but as long as the footage exists, they can rewind and check.

DeltaStreamer's design is built precisely around these two gaps.

Streaming video understanding refers to systems performing understanding and question-answering in real time as video continuously arrives, rather than processing offline after the entire video has ended. This setting imposes rigid constraints on memory and compute: the system cannot keep all historical frames in GPU memory or a context window, and must continuously decide "keep or discard" while observing. This stands in sharp contrast to traditional offline video QA—where the full video is accessible, and retrieval and attention mechanisms can freely revisit any segment. In streaming scenarios, that freedom is stripped away, making compression strategy and memory management the core variables that determine a system's ceiling—and the starting point for works like DeltaStreamer.

Change-Aware Memory: Recording "How It Changed" Rather Than "What's There"

DeltaStreamer's first core component is change-aware memory. Before a query arrives, the agent continuously observes the video stream and explicitly writes visual changes into structured memory.

The key word here is "explicitly." It doesn't just record objects or scenes present in the current frame—it preserves how events unfold step by step. In other words, the memory entries are "changes" themselves, not static snapshots. This way, event transitions aren't swallowed by the compression process, and causal relationships along the timeline are preserved.

This design philosophy is close to human narrative memory: when we recall an experience, we tend to first remember "what happened first, then what happened next," rather than a frame-by-frame inventory of images.

Dense captioning refers to generating natural language descriptions of video frames or segments that cover objects, scenes, actions, and other elements. A common approach is to call a vision-language model (VLM) every few frames to generate a text passage, then use these texts as historical memory. The advantage of this representation is that it's compact and can be fed directly into a language model, but the cost is that it's highly lossy: descriptions typically focus on "what's in the frame," often glossing over the temporal connections between actions and the triggering conditions for state transitions. When a query involves questions like "when did it start moving," dense descriptions cannot provide sufficient temporal resolution. DeltaStreamer's "change awareness" is precisely aimed at this representational deficiency, rewriting memory units from static descriptions into explicit change events.

Query-Driven Planning: First Determine What's Needed, Then Go Get It

When a query arrives, DeltaStreamer doesn't simply retrieve from existing memory—it activates a query-driven planner.

The planner's role is to determine: what evidence is needed to answer this question? Is that evidence already in memory? If not, which segment of historical frames should be revisited?

This step transforms "understanding" from passive matching into active evidence gathering. The planner outputs not an answer, but an evidence acquisition path that guides subsequent memory retrieval and targeted replay.

Targeted Replay: Recovering Lost Details

Targeted evidence gathering with replay is DeltaStreamer's most groundbreaking component.

When the planner finds that descriptions in memory are insufficient to support an answer, the agent revisits historical frames, recovering visual details from raw pixels that weren't retained during the observation phase. The final answer is built on evidence that "goes beyond existing descriptions," rather than relying solely on compressed text memory.

This directly addresses the second flaw mentioned earlier. The replay mechanism essentially opens a backdoor for the system: even if the observation phase made a misjudgment and missed information, as long as the video still exists, there's an opportunity for remediation.

Targeted replay technically requires the system to still access raw video frames or frame indices during the inference stage, rather than retaining only compressed text. This means the underlying system needs to maintain an addressable video store, with the ability to quickly locate and extract relevant segments based on timestamps or event boundaries. Unlike text memory based on retrieval-augmented generation (RAG), replay evidence comes from the pixel level, so it can recover information that was never written into text during the description phase—such as license plate numbers, gesture details, or objects that briefly appeared and disappeared. The cost is additional storage and read overhead, and the planner must accurately identify "which segment to revisit," otherwise replay degenerates into blind scanning.

Stitching Together Continuous Observation and Active Investigation

Looking at the three components together, what DeltaStreamer truly aims to do is bridge two previously separate capabilities: continuous observation and active investigation.

The observation phase handles breadth, using change-aware memory to cover the entire timeline; the investigation phase handles depth, drilling down in a targeted way based on specific questions. The two are connected through the planner, forming a closed loop. The paper describes this as "inspired by how humans observe, remember, and investigate"—and the analogy isn't far-fetched.

Experimental Results

On two streaming video understanding benchmarks, DeltaStreamer achieves improvements over representative methods:

• OVO-Bench: Overall score of 66.07%
• StreamingBench: Overall score of 59.53%

The paper emphasizes that these improvements come from "connecting continuous observation with active investigation for each question," rather than isolated improvements from a single module.

OVO-Bench and StreamingBench are evaluation benchmarks designed in recent years specifically for streaming/online video understanding, distinct from traditional offline video QA datasets. Their core characteristic is simulating "watch and answer as you go": at time step t, the model can only see frames that have arrived so far, yet must answer questions that may involve any historical moment. These benchmarks typically include different task types such as recalling past events, perceiving current states, and predicting future trends, capable of exposing the trade-offs models make between memory retention and timely response. Therefore, score improvements on these two benchmarks are more indicative of true capability in streaming scenarios than improvements on offline datasets, which also explains why the paper uses the results as its core argument.

Signals Worth Watching

DeltaStreamer's value isn't just in the scores. It raises a question worth industry reflection: must streaming video understanding always make a one-time trade-off on "how much to remember"?

Its answer is—no. Let memory record changes, let queries drive evidence gathering, and let replay cover details as a fallback. This combination transforms the irreversibility problem of lossy compression into a retractable retrieval problem. For scenarios requiring long-duration monitoring, real-time QA, and video agents, this approach may be more practical than simply scaling up models.

The project homepage is now public, and interested readers can further explore implementation details.

One-line Summary

DeltaStreamer uses change-aware memory, query-driven planning, and targeted replay to transform the lossy compression challenge of streaming video understanding into a retractable retrieval problem.

Paragraph Summary

Addressing two major pain points in streaming video understanding—blurred event transitions and unrecoverable missing visual evidence—DeltaStreamer proposes a human-like agent framework. It continuously observes and explicitly records visual changes before queries arrive, forming change-aware memory; when a query arrives, a planner determines the required evidence and produces an acquisition path; when existing memory is insufficient, targeted replay recovers details lost during compression from raw historical frames. The three components stitch the breadth of continuous observation with the depth of active investigation into a closed loop. On OVO-Bench and StreamingBench, the method achieves overall scores of 66.07% and 59.53% respectively, outperforming representative methods. The paper emphasizes that improvements stem from the connection between observation and investigation rather than single-module enhancements.