New Breakthrough in Claude Video Understanding: 23-Minute Long Video Processing Costs Only a Few Cents

Most Claude video demos can only handle 15-second clips, because the cost of frame-by-frame input escalates quickly. But a method shared by a developer on Reddit enabled Claude to successfully analyze a full 23-minute video, with the total cost kept within a few cents. This may be the first practical solution for letting AI process videos of real length.

The Bottleneck of Traditional Methods: Frame Cost

The most direct way to make Claude understand a video is to split it into frames and send them to the model one by one. But the problem is that Claude's API bills by input tokens, and image frames consume a large number of tokens. A 15-second short video sampled at 1 frame per second is 15 images; if you increase the sampling rate, the cost grows exponentially.

For meeting recordings, event livestreams, or instructional videos that often run for tens of minutes, this method's cost becomes unacceptable. Many developers therefore give up on using Claude to process long videos, or can only extract clips for analysis.

Tokens and Multimodal Input Cost: In the APIs of large language models such as Claude, both text and images are converted into tokens for billing. Image tokenization is usually based on patches or resolution tiles, and a standard image may consume hundreds to thousands of tokens, far more than text carrying the same amount of information. A video consists of consecutive frames, and even when sampled at a lower frame rate, the total number of tokens grows linearly with video duration. Therefore, when processing long videos, input token cost becomes the main bottleneck. Developers often control costs by lowering resolution, reducing the number of sampled frames, or analyzing only clips, but this sacrifices information completeness. Understanding the token billing mechanism helps evaluate the economics of a "filter first, reason later" strategy.

New Approach: Filter First, Reason Later

This developer (GitHub user DeepAleph) adopted a "two-stage" strategy: first use a lightweight filter to scan the entire video and find key segments, then send the key frames from these segments to Claude for in-depth reasoning.

Specifically, he used the open-source Jev API (a video processing framework) to perform the initial screening. The Jev API analyzes every second of the video, detects signals such as picture changes and motion intensity, and locates moments that may contain important events. In the Tesla event example, the system automatically found the launch moment of the flying Roadster and verified the relevant frames.

Then, only the frames from these few seconds were sent to Claude, and Claude explained the physics behind them. Throughout the process, Claude processed only a small number of key frames rather than all video frames, so the cost dropped significantly.

According to the developer, the total cost was "less than a few cents," and this also included usage of both the Jev API and Claude.

Frame Difference and Optical Flow: The frame difference method detects motion or scene changes by comparing pixel value differences between adjacent frames. It is simple to compute but susceptible to noise; the optical flow method estimates motion vectors of pixels between consecutive frames, can describe the direction and speed of motion more finely, and is often used for action recognition and event detection. The Jev API uses these visual heuristic signals to locate moments "that may contain important events," such as drastic picture changes or fast-moving objects. Such methods do not rely on deep learning, have low operating costs, and are suitable as pre-filters, but their sensitivity depends on threshold settings and may be insensitive to slow or subtle changes.

Technical Details and Limitations

The core of this method lies in the accuracy of "event detection." If the filter misses key segments, Claude cannot analyze them. The Jev API currently relies on visual heuristic methods (such as frame difference and optical flow). It works well for videos with obvious actions and frequent scene changes, but may not be sensitive enough for static dialogue or content with subtle changes.

In addition, the developer did not disclose the specific implementation details and screening thresholds of the Jev API, so reproducing the results requires some debugging. However, the project is already open source on GitHub (repository address: https://github.com/DeepAleph/jevreel), and those interested can study it themselves.

Another limitation is that this method is better suited to "event-driven" video analysis, such as finding the moment when an action occurs or explaining the physical phenomenon at a certain instant. For tasks that require understanding the complete narrative, emotional changes, or long-range dependencies, key frames alone may not be enough.

Event Detection: In video analysis, event detection refers to identifying semantically meaningful moments or segments from a continuous video stream, such as the appearance of an object, the occurrence of an action, or a scene change. Traditional methods are based on frame difference, optical flow, or background modeling, while deep learning methods use 3D convolutional networks or temporal action localization models. The filter in this article belongs to lightweight event detection, aimed at narrowing the scope of subsequent reasoning at low computational cost. The recall rate of event detection (whether key events are missed) directly determines the final analysis performance, so threshold tuning and scenario adaptation are crucial.

Significance for AI Video Applications

For a long time, video understanding has been regarded as the "next big mountain" for multimodal AI. Although models such as GPT-4V and Claude 3 can process images, the cost and latency of video input prevent many application scenarios from being implemented.

This solution provides a pragmatic compromise: instead of pursuing letting the model "watch the entire video," let the model "watch the parts it should watch." This is similar to the attention mechanism when humans watch videos—we do not pay equal attention to every frame, but focus on key events.

If this method is widely adopted, it may spawn a batch of new applications: automatically analyzing meeting recordings and extracting decision points, monitoring abnormal events in surveillance video, automatically summarizing educational videos, and so on. Reducing the cost from "unusable" to "acceptable" is a key step toward commercialization.

Multimodal AI and Video Understanding: Multimodal AI refers to models that can simultaneously process different modal information such as text, images, and audio, such as GPT-4V and Claude 3. Video understanding is an advanced task for multimodal AI, requiring models not only to recognize single-frame content but also to understand temporal dynamics and long-range dependencies. Current models usually treat video as a sequence of images and lack native video encoding capabilities, resulting in high computational costs and difficulty capturing long-term associations. Therefore, industry often adopts the compromise of "key frame extraction + image reasoning" to strike a balance between cost and performance. The practice in this article is a typical representative of this approach.

Reproducibility and Community Reaction

After the Reddit post was published, many users asked about the specific implementation and cost details. The developer said the code is already open source, but users need to configure the Jev API and Claude API keys themselves. Some users pointed out that similar "filter first, reason later" ideas have been studied in academia (such as key frame selection in video question answering), but this is the first time someone has engineered it and publicly disclosed cost data.

Others questioned: if the filter itself is not intelligent enough, will it miss important content? The developer responded that the current method performs well on videos with "obvious climaxes" like the Tesla event, but for more complex video types, the screening strategy still needs further optimization.

Regardless, this project demonstrates a feasible path for long-video AI analysis, and the low cost is surprising. For developers who want to explore video understanding, this is a starting point worth trying.

One-line Summary

A developer used the Jev API to pre-filter key frames, reducing the cost of Claude analysis for a 23-minute video to a few cents.

Paragraph Summary

This article introduces a low-cost solution for letting Claude analyze long videos. The traditional frame-by-frame input method makes it difficult to process long videos because token costs grow linearly with video duration. Developer DeepAleph adopted a two-stage strategy: first use the open-source Jev API to scan the entire video through visual heuristic methods such as frame difference and optical flow to locate key event segments; then send only a small number of key frames to Claude for in-depth reasoning. In the Tesla event video case, the system automatically found the launch moment of the flying Roadster, and the total cost was less than a few cents. This method is suitable for event-driven video analysis, but the accuracy of the filter, sensitivity to static or subtle changes, and long-range narrative understanding remain limitations. The project is open source on GitHub and provides a pragmatic and reproducible starting point for long-video AI analysis.