Qwen3 27B + DeepSeek Harness Real-World Testing: Local Open-Source Agent Performance Evaluation

The capability boundaries of open-source large models are being continuously pushed. Recently, a tech blogger shared a complete real-world test on Bilibili combining Alibaba's Qwen3 27B model with the DeepSeek Harness agent framework, with impressive results—this combination not only runs smoothly on consumer-grade hardware but also demonstrates some "hidden capabilities" rarely seen in open-source vision-language models. This article, based on that test content, covers the deployment approach, performance results, and key pitfalls.

Qwen3 27B Model Positioning: Strong Performance in a Small Package

Qwen3 27B is currently one of the most powerful local models that can run on consumer-grade hardware, with particularly outstanding cost-effectiveness. According to the Artificial Intelligence Agentic Index referenced by the blogger, this model scores approximately 51 points, only slightly below Kimi K2 with its roughly 2.8 trillion parameters, while Qwen3 27B is merely a 27-billion-parameter dense model.

Notably, Qwen3 27B uses a Dense architecture, meaning the model activates all 27 billion parameters for computation during each inference pass. In contrast, Mixture of Experts (MoE) architectures like DeepSeek-V3 or Kimi K2 may have total parameter counts reaching trillions, but only activate a small subset of expert networks during each inference. The advantage of dense models lies in simpler implementation, more predictable inference latency, and typically higher parameter utilization efficiency given the same number of active parameters; the disadvantage is the inability to reduce per-inference computational overhead through sparse activation like MoE. The fact that Qwen3 27B can approach the performance of trillion-parameter MoE models with just 27 billion parameters speaks volumes about the optimization level of its architecture and training data.

Even more notably, its overall intelligence performance is very close to Claude 4.x (Max setting), which was considered SOTA just months ago. The blogger also acknowledged that benchmark scores inevitably carry some "benchmark gaming" suspicion, but emphasized that the model's performance in real-world scenario testing is genuinely solid—which is the core highlight of this evaluation.

Local Deployment Setup and Hardware Performance

The model is already available on Hugging Face, and the blogger provided clear deployment recommendations:

• Apple Silicon devices: Recommended to use the MLX version
• Linux / Windows: Recommended vLLM or SGLang inference solutions

This test ran on an NVIDIA DGX Spark dual-node cluster using the NV FP4 quantized version. FP4 (4-bit floating point) quantization is an ultra-low precision numerical format from NVIDIA that compresses model weights from standard FP16 (16-bit floating point) to just 4-bit floating point representation. Each weight value is compressed from 16 bits to 4 bits, theoretically reducing model VRAM usage by approximately 75%. Unlike common INT4 integer quantization, FP4 retains the exponent-mantissa structure of floating-point numbers, better representing the distribution of small values near zero, thus maintaining higher model accuracy at extremely low bit widths. For a 27B parameter model, FP16 requires approximately 54GB of VRAM, while FP4 quantization reduces this to roughly 13.5GB, making it possible to run on consumer GPUs or compact inference hardware like the DGX Spark.

It's worth noting that the DGX Spark wasn't designed for running 27B dense models, so single-thread speed is approximately 15–20 tokens/second, while concurrent scenarios can reach 60–70 tokens/second, which is overall in usable territory. During deployment, you can also enable speculative decoding, where the framework automatically downloads a matching drafter model to accelerate inference.

Speculative decoding is an important technique for accelerating large language model inference. Its core idea is: use a lightweight "drafter model" to quickly generate multiple candidate token sequences, then have the target large model verify the correctness of these candidate tokens in parallel. Since the parallel computation cost of the large model verifying multiple tokens is far lower than the serial cost of generating tokens one by one, overall inference speed can improve 2-3x when the drafter model's prediction accuracy is high. The key to this technique lies in the distribution matching between the drafter and target models—the higher the match, the more tokens are accepted, and the more pronounced the acceleration effect. DeepSeek Harness's design of automatically downloading matching drafter models greatly lowers the barrier for users to configure speculative decoding.

Visual Understanding Capabilities: A Language Model That Can "Draw Boxes"

The most impressive part of this evaluation comes from Qwen3 27B's multimodal visual capabilities. This goes far beyond simply recognizing image content.

The blogger first provided a reference image of a raccoon and asked the model to describe the content. The model not only gave an accurate description but also autonomously decided to call OpenCV and write its own analysis script, performing a series of computer vision operations including edge detection and corner recognition—the entire process running locally on the DGX cluster without any external tools.

Precise Counting and Object Localization Tests

Vision-language models have historically performed poorly on "counting" tasks, but Qwen3 27B delivers a pleasant surprise. When asked how many cars are in an image, the model gave an answer of 26 vehicles, counting by zones row by row (11 in the top row, 9 in the main parking area, etc.). From the chain-of-thought, we can see that the model is able to segment the image into variable-sized patches, identifying and counting objects within each patch—a capability mentioned in DeepSeek's vision papers but rarely seen in actual products.

In vision-language models, images are typically segmented into fixed-size patches (such as 16×16 or 14×14 pixel blocks in ViT) before being fed to the visual encoder. However, the Qwen3 series introduces a dynamic resolution processing mechanism—the model can segment images into patches of different sizes and quantities based on actual image content and task requirements. This approach stems from research ideas like NaViT (Native Resolution ViT): for regions requiring fine-grained recognition (like densely arranged vehicles in a parking lot), smaller and more numerous patches can be used for higher resolution; for unimportant regions like backgrounds, larger patches save computational resources. This adaptive mechanism is precisely why Qwen3 excels at counting tasks—it can allocate more visual attention to regions requiring precise counting.

Going further, the blogger asked the model to draw bounding boxes around objects in the image. The model once again autonomously wrote a Python program, with bounding box coordinates coming directly from the model's own pointing capability. The results were quite impressive: although some boxes were slightly off (e.g., covering too large an area, questionable pickup truck classifications), the localization accuracy for multiple objects in different orientations is remarkably rare among open-source vision-language models.

Bounding box drawing capability is known in academia as Visual Grounding—meaning the model can not only understand image content but also precisely output the spatial coordinates of target objects in the image (typically in [x_min, y_min, x_max, y_max] format). This capability requires the model to encounter large amounts of spatially annotated data during training (such as COCO, RefCOCO datasets) and establish precise mappings between language descriptions and pixel space. Traditionally, this requires specialized object detection models (such as YOLO, Faster R-CNN). When a general-purpose language model possesses this capability, it means it can autonomously complete the "observe-locate-act" loop in agent workflows without relying on additional visual toolchains—a qualitative improvement in autonomous agent practicality.

Reasoning Intensity Settings: A Double-Edged Sword

Qwen3 27B offers four reasoning intensity levels: Off, Low, Medium, and Ultra-High (note: there is no "High" level). This setting directly determines output quality but also hides a major pitfall.

The blogger specifically warns: be very careful when choosing reasoning intensity. At "Ultra-High" or even "Medium" settings, the model is highly prone to overthinking, sometimes consuming the entire completion budget on the thinking process and ultimately producing no output at all—a situation that occurred repeatedly during testing.

Qwen3's multi-level reasoning intensity essentially controls the depth and breadth of internal reasoning (Chain-of-Thought) the model performs before generating its final answer. When set to "Ultra-High," the model is allowed (even encouraged) to perform extensive self-reflection, approach comparison, and logical verification, all of which consume output token quota. The root cause of the "overthinking" problem is that during reinforcement learning training, the model was rewarded for "thinking more deeply," but there's no effective mechanism to determine when it should stop thinking and output a conclusion. This causes the model to potentially fall into infinite self-questioning loops on complex tasks, exhausting the entire context window without producing effective output. This is also a common engineering challenge facing current reasoning models (such as OpenAI o1 series, DeepSeek-R1).

Web Generation Comparison Across Different Reasoning Intensities

Using "create a website about itself using web search" as an example, the results across four reasoning levels differ dramatically:

• Off: Outputs approximately 20K tokens, resulting in typical small-model "AI slop," even fabricating data
• Low: Generation quality jumps dramatically, with beautiful visual effects—the blogger considers it even better than early GPT-5.x
• Medium: Adds smooth animations with significantly improved quality, but token consumption increases substantially
• Ultra-High: Best visual quality, but with noticeably different color schemes from Low/Medium

Interestingly, the blogger also had the model generate a real-time International Space Station tracker that updates strictly every 5 seconds as requested—one of the best results he's ever seen from an open-source model.

Token Consumption Data

The measured token consumption provides valuable reference:

Reasoning Intensity	Input Tokens	Output Tokens	Duration
Off	82K	20K	~15 min
Low	170K	31K	~35 min
Medium	860K	38K	~63 min
Ultra-High	~500K	-	~86 min

Clearly, once reasoning intensity is increased, token consumption grows "explosively," requiring a balance between quality and cost. Notably, the dramatic inflation of "Input Tokens" in the table primarily comes from the model's chain-of-thought content—in multi-turn agent interactions, previous thinking processes accumulate in the context window, causing the input token count for each conversation turn to snowball.

DeepSeek Harness: An Agent Framework with Auditable Trajectories

The other protagonist of this evaluation is DeepSeek Harness. Its core design philosophy is "everything is a plugin," which the blogger highly praises.

This highly modular agent framework architecture means that tool calls (such as web search, code execution, file operations), memory management, planning strategies, and other functions are all encapsulated as independent plugin modules that interact with the core scheduler through a unified interface protocol. Developers can flexibly combine different capability modules without modifying the framework's core code, achieving true "plug-and-play."

The standout feature is Trajectory: click to view everything that happened in the context—initial system prompt, user messages, available skill list, every action the model took, tool call payloads, schemas, return results, and even the timeline. This fully auditable transparency typically requires additional tracing tools to achieve, but Harness provides it natively, making it easy for developers to identify agent failure points and iterate on tools.

In production environments, this trajectory audit capability is crucial for debugging agent "hallucination decisions" or "tool misuse." Traditional approaches require integrating third-party observability platforms like LangSmith or Arize Phoenix to achieve similar effects, while DeepSeek Harness builds this capability directly into the framework itself, significantly reducing development and operational complexity.

Worth mentioning is that DeepSeek Harness surpassed 160K GitHub Stars within less than a week of release, called by the blogger the fastest-growing open-source project in GitHub history. Interestingly, the project does not accept any external contributions—all code is completed by the DeepSeek team.

Conclusion: A New Option for Local Open-Source Agents

The combination of Qwen3 27B and DeepSeek Harness represents a new high point for local open-source AI agents. The former achieves near-frontier closed-source model capabilities with just 27 billion parameters, with particularly impressive performance in visual counting and bounding box drawing; the latter provides a transparent, pluggable, and auditable agent runtime environment. For developers looking to build autonomous agent systems on local hardware, this is a combination worth exploring in depth—but be sure to watch out for the token costs and overthinking risks that come with reasoning intensity settings.