CapMem: A New Benchmark for Long-Video Episodic Memory Using Text Captions

The Memory Problem Facing Wearable Assistants

With the rise of smart glasses, AR assistants, and other wearable devices, enabling AI to remember and understand long egocentric videos captured from a user's first-person perspective has become a core technical challenge. An ideal wearable assistant should possess episodic memory — the ability to recall scene details from minutes or even tens of minutes earlier in the user's experience.

The reality, however, is sobering. Current mainstream vision-language models (VLMs) face three major bottlenecks when processing long videos: limited frame budgets, rapidly escalating computational costs as visual tokens grow, and retrieval failures in extremely long contexts. When videos routinely stretch to tens of minutes, feeding all frames directly into a model is neither economical nor reliable.

When VLMs process video, they typically convert video frames into "visual tokens" as model input. Each frame can correspond to hundreds of tokens — a 30-minute video sampled at one frame per second would produce tens of thousands of visual tokens, far exceeding the context window limits of most current models (typically 4K to 128K tokens). Even models that support ultra-long contexts face quadratically scaling attention computation as token counts grow, causing inference costs to skyrocket. Furthermore, research has shown that models exhibit a "lost-in-the-middle" phenomenon in very long contexts — retrieval accuracy for information located in the middle of a sequence drops significantly. These three constraints together explain why brute-forcing all frames is neither feasible nor reliable in real-world wearable scenarios.

Using Captions as Reusable Memory

CapMem introduces a clever approach: rather than repeatedly processing heavy visual information, convert video content into text captions and use them as a reusable episodic memory store.

Text has natural advantages over raw video frames — it consumes far fewer tokens, is easy to store and retrieve, and can be reused across multiple queries. Based on this insight, the researchers define a new task: Episodic Memory Video Caption QA, which asks: can AI effectively perform episodic reasoning over long videos using captions alone?

The CapMem Benchmark

To systematically validate this hypothesis, the research team constructed the CapMem benchmark dataset, notable for both its scale and annotation quality:

• 75 videos totaling 33.7 hours
• 1,000 multiple-choice questions, all human-annotated
• Coverage across 16 different scenarios

This scale ensures the evaluation reflects real-world diversity rather than being confined to a single task type.

Experimental Results: Caption-Based Memory Actually Works

The most striking findings center on long videos (over 20 minutes).

In this category, the full-coverage CaptionQA approach — generating captions using 30-second and 60-second windows — outperforms conventional direct VideoQA. Specifically:

• Under the 30-second caption window, 10 out of 12 models performed better
• Under the 60-second caption window, 8 out of 12 models performed better

To rule out the possibility that the caption approach simply benefits from seeing more frames, the researchers conducted matched-frame controlled experiments across six Qwen-series models. Even under fair, frame-count-aligned conditions, the caption approach maintained average accuracy gains of 3.22 and 2.55 points, respectively. This confirms that the performance improvement stems from the caption-based memory representation itself, not from additional visual input.

Retrieve-and-Verify Further Boosts Performance

The research team also designed a caption-guided retrieve-and-verify framework. This mechanism first retrieves relevant segments from the caption memory based on the question, then performs verification reasoning — further boosting accuracy by up to 5.3 points.

This result demonstrates that captions can serve not only as static memory storage, but also as the foundation for smarter retrieval strategies, forming a complete long-video reasoning pipeline.

Retrieval-Augmented Generation (RAG) is the core inspiration behind this framework. The fundamental logic of RAG is to avoid sending all information into the model at once; instead, vector retrieval or keyword matching is used to find the most relevant segments for the current query, and only that condensed context is passed to the model for reasoning. In the long-video setting, captions are natural retrieval units — each caption corresponds to a fixed time window, is semantically self-contained, and is far smaller in size than the corresponding video frames. The "retrieval" step addresses the localization problem of "where to find the answer," while the "verification" step improves accuracy by performing secondary reasoning over candidate answers. Together, they effectively reduce the noise that arises when reasoning directly over full caption sequences.

Significance and Outlook

CapMem's value lies in offering a pragmatic solution to a long-standing engineering constraint. Under the real-world limitations of wearable device compute and memory, "compressing" visual information into text captions both reduces cost and sidesteps the pain points of long-context retrieval.

From a broader perspective, this work signals to the field that handling long videos doesn't necessarily require ever-larger visual token budgets and ever-longer context windows. Converting modalities into more compact textual representations, combined with retrieval-augmented mechanisms, may be a more efficient path toward practical episodic memory assistants.

For teams exploring smart glasses, AI assistants, and similar products, both the benchmark and the methodology offered by CapMem provide directly actionable reference points. Of course, the caption-based approach has inherent limitations — the quality of caption generation itself, and visual details that cannot be precisely described in language, may both become information bottlenecks. These are challenges that future research will need to address.