Qwen3 27B Local Benchmark: This Small Model Outperforms Opus on Agentic Tasks

Open-source models continue to evolve faster than anyone expects. When the Qwen team unveiled their 2.4-trillion-parameter flagship closed-source model, they also promised to release a 27B open-weight version — and that's the one most developers actually care about. Nobody can run a 2.4T-parameter monster at home, but a 27B model? That fits in your GPU.

That model is here now. The weights are up on Hugging Face under an Apache 2.0 license. A well-known AI benchmark blogger abroad ran a deep evaluation and came back with a surprising conclusion: this 27B model, running entirely on local hardware, outperforms the flagship closed-source Opus 4.6 Max on several real-world agentic tasks.

A Brain Built for Local Agents

The blogger opened with an important clarification: he didn't include this model in his regular leaderboard (KingBench). Not because he forgot or ran out of time, but because "this model simply wasn't built for my leaderboard."

His benchmark is designed to separate top frontier models — hard reasoning problems, tricky counting tasks, long-horizon agentic challenges — to find the ceiling of models like Opus, GPT-5.6, and Qwen3 Max. Throwing a 27B open-source model into that mix would naturally yield a low score, leading people to wrongly conclude the model is weak. That would be completely wrong.

Because this model isn't trying to win a "reasoning beauty contest" — it's meant to be a great local agent brain. That's an entirely different job.

Architecturally, it's built on the Qwen3.5 base and is a native vision-language model, capable of natively understanding images and video — from STEM diagrams and documents to long-form video. The architecture uses a hybrid design: 64 layers, mostly Gated DeltaNet modules with periodic Gated Attention layers mixed in. This allows it to handle very long contexts without blowing up memory. The native context length is 262,144 tokens, extendable to a full 1 million tokens via RoPE extension techniques like YARN — almost absurd for a 27B model, putting it in the same league as frontier hosted models, but running on your own machine.

Gated DeltaNet is a variant of linear attention, distinct from the standard self-attention used in traditional Transformers. Standard self-attention scales quadratically with sequence length (O(n²)), meaning memory consumption becomes unmanageable at hundreds of thousands of tokens. DeltaNet approximates attention using the Delta Rule (an update mechanism from associative memory research), compressing complexity to linear (O(n)). The "gated" part adds a learnable gating mechanism, letting the model dynamically decide which historical information to retain and which to discard — unlike early RNNs that crudely blended all history. Qwen3 27B's 64-layer design uses DeltaNet as the backbone with periodic standard attention layers inserted at key positions. This hybrid design aims to combine the best of both: linear layers ensure memory efficiency at long sequences, while standard attention layers provide precise global information retrieval where it matters most. This is the architectural foundation that lets it handle 262K — and even 1M — token contexts on local hardware.

Surprising Numbers on Agentic Tasks

The benchmark results are what really make you sit up. On SWE-bench Pro, it scores 61.7 — compared to 57.6 for Qwen3.7+ and just 53.4 for Opus 4.6 Max. A locally-run 27B model beating a frontier hosted model on real-world software engineering tasks. That's not a typo.

More data points: Terminal Bench 2.1 jumps from 63.4 (previous-gen 27B) to 73.0; its own Qwen SWE-bench goes from 49.3 to 79.0; Deep SWE 1.1 leaps from 13.3 to 42.2 — a generational improvement that borders on ridiculous. On CoworkBench it scores 70.7, beating Opus's 68.2. Instruction following on IF-Bench comes in at 79.5 versus Opus's 62.5. LiveCode Bench V6 scores 90.3, topping the entire comparison.

On pure reasoning, it's honest about the gaps. GPQA Diamond at 89.2 is respectable but below Opus, and HLE is only 30.8 versus Opus's 40.0. The blogger sees this gap as expected — it's a "frontier-adjacent agent," not a "frontier reasoner."

The most surprising results are in vision and computer operation. On OS World Verified, a real computer operation benchmark, it scores 84.3 versus 73.3 for Qwen3.7+ and 72.7 for Opus 4.6 Max — making this 27B local model the strongest computer-operation model in the entire comparison. Browser navigation on Web Arena goes from 48.5 to 64.8. Mobile Android World reaches 81.9. On multimodal software engineering (SWMM), it scores 38.6 versus Opus's 27.1. This is a model that can see a screen, understand what's on it, and actually interact with it — 27B, local, Apache 2.0.

SWE-bench is one of the most respected real-world benchmarks for software engineering capability. It requires models to fix issues from actual GitHub repositories — given a codebase and problem description, the model must locate the bug, write a patch, and pass a test suite. SWE-bench Pro is a stricter upgrade with more complex multi-file modification tasks and more rigorous test validation, making it harder to game. OS World specifically evaluates a model's GUI operation ability in real operating system environments — identifying UI elements from screenshots, executing clicks, typing, dragging, and completing multi-step cross-application tasks. This is fundamentally different from "describing how to do something"; it tests whether a model can genuinely take control of a mouse and keyboard. Both benchmarks are valuable precisely because they measure real task completion rates in real work scenarios, not multiple-choice questions or code completion.

Thinking Control and Deployment

For agentic workflows, thinking control is critical. The model offers three levels of reasoning intensity: X-high (default, for complex tasks requiring deep analysis), medium (balanced accuracy and speed), and low (optimized for speed and cost). Thinking mode is on by default and can be disabled per-request via the enable_thinking parameter.

There's also a preserve_thinking parameter that retains reasoning context across multi-turn conversations rather than discarding it each time. Anyone who has watched a local model forget its own plan midway through a multi-step task will understand why this matters. The blogger's advice: use low or medium for mechanical, repetitive agent steps, and save X-high for the parts that actually require thinking — otherwise you'll waste time watching it "think about whether to read a file."

On deployment, the blogger breaks from his usual approach and recommends Ollama and LM Studio over VLLM. "For most people, this is the right answer — I was overcomplicating things before." VLLM makes sense for multi-GPU machines serving a team, but for a single person running agents on a single machine, it just adds Python environment pain with no real benefit.

The good news is the model received day-one support on both platforms. Hugging Face already has around 278 quantized variants, including GGUFs from Unsloth, Bartowski, GGML.org, and the LM Studio community, plus 4-bit and 8-bit MLX builds for Mac users.

YARN (Yet Another RoPE extensioN) is a technique for extending the positional encoding range of large language models. RoPE (Rotary Position Embedding) is the dominant positional encoding scheme in modern LLMs, but models trained on fixed-length sequences suffer steep performance drops when inference exceeds that training length. YARN applies differentiated scaling to positional encoding components at different frequencies, allowing the model to generalize far beyond its training length while preserving performance within that range. Compared to earlier approaches like linear interpolation or NTK-aware scaling, YARN delivers more stable perplexity and task completion on long texts. Qwen3 27B's native training context is 262,144 tokens; with YARN it extends to 1 million tokens — though retrieval quality typically degrades with distance at very long contexts, and not all information can be utilized equally effectively.

Ollama and LM Studio: Practical Notes

Ollama is the simplest option. The model is in the official library — just run ollama pull qwen3:27b. The default tag is about 18GB, supports 256k context, and retains both text and vision capabilities (vision models often get quantized down to text-only, losing half their value). Apple Silicon users should use the qwen3:27b-mlx tag for significantly better performance than generic GGUF.

The blogger stresses one "most important thing": set the context length in Ollama. Ollama doesn't automatically use the model's full context — it uses whatever num_ctx is configured, and the default is tiny. You might pull a model supporting 256K tokens and run it with a few-thousand-token window, then wonder why your agent forgets what it's doing after three steps. "It's like buying a sports car and driving in first gear the whole time."

LM Studio suits users who want a GUI and more control. It tells you before downloading whether a given quantization fits your hardware — extremely helpful for newcomers. When loading, you can set context length, GPU layer offloading, and enable Flash Attention. When VRAM is tight, KV Cache quantization can drop the cache to 8-bit, trading a tiny quality hit for a much larger usable context — almost always a good deal for agentic work. Note that LM Studio uses port 1234, not Ollama's 11434 — mixing them up is a classic five-minute debug trap.

Sampling parameters also matter: for thinking mode, use temperature 1.0, top-p 0.95, top-k 20; for non-thinking instruct mode, use temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5. Default sampling settings will noticeably degrade performance, and you'll blame the model unfairly.

KV Cache (Key-Value Cache) is the core inference acceleration mechanism in Transformers: when generating each new token, the model needs to access the Key and Value vectors for all previous tokens. KV Cache stores these in VRAM to avoid redundant computation. The problem is that KV Cache memory grows linearly with context length — for a 262K-context model, this overhead can far exceed the model weights themselves. KV Cache quantization (e.g., dropping to 8-bit or 4-bit) reduces VRAM usage by lowering the numerical precision of these vectors, at the cost of small computational errors. For agentic workloads, this trade-off is usually worth it: agent tasks depend more on the model "remembering" context across a long workflow (requiring a large window) than on precise numerical computation at each step. Trading slight precision loss for a larger usable context window typically improves end-to-end task completion rates.

Hermes Agent: A Complete Local Agentic Stack

The heart of the video is Hermes Agent — an open-source agent framework with persistent memory, vendor routing, sub-agents, message integration, and no vendor lock-in.

Hermes lives and dies by tool calling. Its entire purpose is enabling agents to actually do things. If the model can't call tools cleanly, Hermes degrades into a fancy chatbot. Qwen3 27B's core selling point is "stronger autonomous planning and better environment feedback handling for more reliable end-to-end task completion" — exactly what Hermes needs.

Worth noting: both Ollama and LM Studio handle tool call parsing natively, so there's no need to manually pass parser flags or wrestle with Jinja templates. Hermes sub-agents also inherit the parent model configuration, avoiding vendor-mismatch issues mid-task on long runs.

Put it all together: a 27B model running on your own hardware, 262K token context, native vision for screenshots and charts, working tool calling (OS World 84.3, Web Arena 64.8), with Hermes handling orchestration, memory, and sub-agents. That's a complete local agentic stack — and every token costs nothing.

Where the Limits Are

The blogger is candid about imperfections. The reasoning ceiling is real. HLE at 30.8 means that if you throw a genuinely hard novel problem at it, it will struggle where frontier models cruise through. He wouldn't use it for architectural decisions or tricky debugging.

But that's not what it's for. Its battlefield is "lots of work that's not hard, just long": reading files, running commands, checking output, operating a UI, filling forms, following a plan across 50 steps without losing the thread. That's the real work — and this model is very good at it.

The blogger's final recommendation is simple: use Ollama if you want the easiest path, use LM Studio if you want a GUI and quantization control, and in either case point Hermes at the local endpoint and configure your context length. "Qwen has consistently done one thing — released sizes that regular people can actually run, and they always punch above their weight. Beating Opus 4.6 Max on SWE-bench Pro, OS World, CoworkBench, and IF-Bench with 27B parameters and an Apache 2.0 license is not a small thing."