DeepSeek New Flash Benchmarked: The Most Capable Open-Source Model for Local Visual Reasoning

DeepSeek's newly released Flash model has been praised by many testers as one of the most impressive releases from the team in recent memory. It pushes efficiency to the extreme while natively supporting visual detection, image reasoning, and object counting — meaning you no longer need to wire up a separate computer vision pipeline. This article is based on an in-depth hands-on evaluation of the model, summarizing its real-world performance and practical use cases.

Architecture Overhaul and Remarkable Efficiency Gains

This generation of Flash is not a simple iteration — it's a complete architectural redesign relative to the previous Flash. The model is nearly twice the size of its predecessor, yet efficiency has improved dramatically. Testers specifically highlighted the KV cache optimization: compared to the V1 model, KV cache efficiency improved by approximately 337× — a genuinely staggering number.

This shows up most concretely in cache hit rates during long-running tasks. In one case shown by the tester, the cache hit rate reached 99.7% — out of roughly 40 million total tokens generated, nearly 39 million were cache hits. This means that when running long-horizon agentic tasks, users effectively pay cached pricing rather than standard input pricing, dramatically cutting costs. API calls can also reach approximately 300 tokens/second.

From a pricing perspective, the tester flatly called this one of the best value-for-money models currently available. While it's not ideal as an orchestrator, it's an excellent choice as a sub-agent or for local deployment.

KV Cache (Key-Value Cache) is a core optimization mechanism in LLM inference. In the Transformer architecture, generating each new token requires recomputing the Key and Value matrices in the attention mechanism over all previous tokens. The KV cache stores these results so subsequent steps can reuse them directly, avoiding redundant computation. In long-context tasks, higher cache hit rates mean less actual computation, significantly reducing both cost and latency. A 337× efficiency improvement means the model needs to read and write far less cache data when processing the same context length — especially critical for agentic tasks that repeatedly access long documents or conversation histories, where cumulative token counts can reach hundreds of thousands and cache efficiency directly determines whether running costs are viable.

The Harness You Choose Sets Your Performance Ceiling

One point repeatedly emphasized throughout the testing is: the harness (execution framework) you use to run the model will massively affect the final results. This is true for all models, but it's especially pronounced with DeepSeek.

The tester cited official DeepSeek experimental data on Terminal Bench 2.0: the same DeepSeek v1.1 model scored anywhere from 69.8% to 74.2% depending on the harness — a gap of nearly 5 percentage points. Choosing the wrong harness leads to suboptimal results.

The tester primarily used DeepSeek's official "DeepSeek Harness" and ran it locally on two DGX machines. He considers it one of the best execution frameworks for DeepSeek models, with its core advantage being a built-in closed-loop flow of plan → implement → visual verify → iterate.

Harness (Execution Framework) in the AI agent context refers to the complete engineering scaffold wrapping model API calls — typically covering prompt templates, tool-calling logic, error retry mechanisms, multi-step task orchestration, and context management. The same underlying model behaves in fundamentally different ways under different harnesses — for example, whether it has tool-calling capability, whether it supports multi-round planning, and whether it can receive and process intermediate results. Terminal Bench 2.0 is a benchmark specifically designed to evaluate models on complex multi-step tasks in terminal environments, covering real engineering scenarios like code execution and file operations, and it is highly sensitive to harness design. This is why benchmark numbers are not directly comparable when different execution frameworks are used.

Visual Generation: 3D Rendering That Validates Itself

The model's visual generation capability is a standout upgrade in this release. The tester's first experiment was fetching the real-time position of the International Space Station and rendering it onto a 3D globe. The result was not only high quality visually, but accurately reproduced the day/night distribution — correctly showing North America in darkness and Asia in daylight.

The key mechanism behind this: the DeepSeek Harness has the model first implement, then visually verify via browser, because the model can now accept visual input and iterate based on it. The tester observed this has become the fixed pattern for this harness — plan, implement, verify, and loop.

Other generation examples included scene rendering with clouds and adjustable time of day, a sea turtle swimming underwater, and a stylish 1-bit aesthetic visual. Compared to the previous Flash generation — which could barely produce decent visual designs — this generation represents a qualitative leap under the right harness.

AutoML: The First Open-Weight Model That's Genuinely Good at Training ML Models

The tester described this as "the first open-weight model that's truly good at training machine learning models." He provided a dataset with only a brief description and asked the model to make its own decisions without further clarification.

The model then independently formulated a plan, performed feature selection, and ran multiple experiments with different hyperparameters to identify the best model — a workflow that closely mirrors what a human ML engineer would do. More interestingly, the tester deliberately included an ambiguous feature with a data leakage trap, and the model proactively identified and removed it. The tester mentioned he plans to produce a dedicated deep-dive video on using LLMs for AutoML.

Data Leakage is a serious modeling error in machine learning where the training set contains information that wouldn't be available during real-world prediction, causing the model to show inflated performance on test sets while performing poorly in deployment. Common forms include: mixing future information about the prediction target into features, normalizing on the full dataset before splitting into train/test sets, and using highly correlated proxy variables. Deliberately constructing a dataset with a leakage feature is a fairly high-bar trap — many junior engineers would miss it too. The model's ability to proactively identify and remove the offending feature suggests it's not just mechanically executing a modeling pipeline, but has developed a degree of data quality awareness — which is one of the core reasons the tester called it "the first open-weight model truly good at AutoML."

Native Visual Analysis: Strong at Extraction, Weak at Localization

Native visual reasoning is the capability the tester values most for practical use. He is actively building a benchmark to evaluate vision-language models and tested this directly via API calls (not in a harness environment) with single-shot predictions.

For automatically extracting component dimensions from technical drawings, the model performed surprisingly well. The tester repeatedly stressed that the prompt itself dramatically changes outcomes and requires careful design.

There is one clear weakness, however: bounding boxes frequently appear offset, with the box not precisely enclosing the target object. Interestingly, despite the imprecise box positions, the actually extracted data (such as the annotated number "20") was accurate. This is worth keeping in mind if you plan to use it for automated evaluation pipelines.

For object counting and reasoning, the model performed strongly. The tester asked it to detect all watercraft and classify them — handled with ease. A follow-up question asking "how many boats are not sailboats, answer with a single integer" was also answered correctly after reasoning over the image.

Bounding Box is a rectangular region used in computer vision to localize target objects, typically expressed as top-left coordinates plus width/height, or as two corner points. In vision-language models, generating accurate bounding boxes is a distinct capability — it requires the model to simultaneously get semantic understanding ("this is a door") and precise spatial coordinate output (four numerical values in the image coordinate system) right. These rely on different capability pathways: semantic recognition primarily depends on vision-language alignment training, while precise localization depends more on specialized detection heads or coordinate regression training data. Flash's pattern of "semantically correct but spatially offset" bounding boxes is a textbook example of the uneven development of these two capabilities in current multimodal LLMs. For use cases where detection results feed into downstream automated pipelines (such as cropping, measurement, or re-annotation), additional post-processing steps will be needed to correct coordinate offsets.

Positioning and Target Use Cases

The tester acknowledged that this video doesn't discuss benchmark scores, because he believes the model doesn't match top-tier models like Opus 5 in overall capability. But his core point is: not every application needs SOTA performance.

For visual analysis use cases, this is an extremely affordable option with sufficient accuracy. For long-running agents, it performs very capably as a sub-agent — especially under the right harness. Factoring in pricing, whether deployed locally or via API, it's a highly competitive option right now.