Ornith 35B vs Qwen 27B Benchmark: Full Local Deployment Performance Evaluation

The debate over open-source LLM performance never really ends. When the DeepSeek team released the Ornith 1.5 series, many developers were curious: could this 35B MoE architecture — claiming to rival commercial models — outperform the already battle-tested Qwen 2.5 27B on real hardware? One developer ran a complete head-to-head comparison on an AMD RX 7900 GPU, and the results were surprising.

Ornith 1.5's Three-Stage Training Innovation

Ornith 1.5 uses a unique three-stage reinforcement learning architecture. In the first stage, a task generator allows the model to autonomously analyze codebases and generate training tasks — earning rewards based on correctness, difficulty, and novelty — effectively avoiding repetitive training. The second stage builds an execution framework and safety checking mechanism. The third stage synthesizes feedback from the first two to produce the final output.

This self-evolving training approach has led to significant improvements on coding benchmarks like SWE-bench. SWE-bench, released in 2023 by Princeton University's NLP group, is a software engineering capability benchmark comprising 2,294 GitHub Issues and their corresponding fix patches drawn from 12 real-world Python open-source projects including Django, scikit-learn, and sympy. The benchmark requires a model — given an issue description and the full codebase — to autonomously locate the problem, understand the context, and generate a correct fix. Because the tasks come from real engineering scenarios rather than artificially constructed programming exercises, SWE-bench is widely regarded as the gold standard for measuring an AI coding assistant's real-world capability.

Official figures for the 35B version show a score of approximately 68 on SWE-bench, while the lightweight 9B version even reaches 71. In DeepSeek's internal testing, Ornith 1.5 jumped from 8 points in version 1.0 to 56 points, approaching the current best of 59.

Real Hardware Testing: Speed vs. Accuracy

The test environment used an AMD RX 7900 (24GB VRAM) + Ryzen 9 processor + 64GB RAM, running the in-house Benchy 0.4 test suite. This suite contains 21 tests covering classification, instruction following, tool calling, context retrieval, code generation, and more.

Ornith 35B Performance

• Overall accuracy: 85% (18/21 tests passed)
• Generation speed: 74 tokens/sec
• Prompt processing: 850 tokens/sec
• VRAM usage: 20GB
• Total time: ~26 minutes

It's worth noting that generation speed (decoding speed) and prompt processing speed (prefill speed) are fundamentally different performance metrics. The prefill phase is a compute-intensive operation where the model processes all input tokens in parallel and builds the KV cache — primarily limited by GPU compute. The decoding phase generates output tokens one at a time and is memory-bandwidth-bound, primarily constrained by VRAM bandwidth. Ornith's drastically higher prompt processing speed (850 t/s vs. Qwen's 240 t/s) is directly attributable to the MoE architecture activating only a fraction of parameters at each step, dramatically reducing compute during prefill. The narrower gap in decoding speed (74 vs. 58 t/s) reflects that the bottleneck there shifts to memory access rather than computation.

Ornith performed flawlessly on basic tasks including classification, instruction following, and tool calling, with well-formed XML output and perfect context retrieval scores. In game code generation tests, it successfully produced multiple runnable HTML5 applications — including an asteroid physics engine, Breakout, and Conway's Game of Life — achieving 99% accuracy.

However, Ornith showed serious weaknesses in complex reasoning scenarios. On a dependency scheduling task, the model got stuck in a self-correction loop, generating 18,000 reasoning tokens over 7 minutes without producing valid code. On a cybersecurity CTF challenge, it wrote more than 17,000 tokens of detailed analysis but missed the critical breakthrough. CTF (Capture The Flag) is the most popular skill competition format in cybersecurity, requiring participants to find vulnerabilities, break encryption, or perform reverse engineering in simulated scenarios. Introducing CTF tests into AI evaluation is valuable precisely because these tasks demand multi-step reasoning, creative thinking, and cross-domain knowledge integration — the model must not only understand code logic but also recognize security vulnerability patterns and construct exploits, making them a far more demanding test of deep reasoning than standard code generation. Ornith's failures here reflect a stability issue with MoE architectures when handling multi-step reasoning chains.

Qwen 2.5 27B: A Clear Stability Advantage

Under the same test conditions, Qwen 2.5 27B (Q4 quantized, 16GB VRAM) showed a markedly different profile:

• Overall accuracy: 99% (20/21 tests passed)
• Generation speed: 58 tokens/sec
• Prompt processing: 240 tokens/sec
• VRAM usage: 16GB
• Total time: ~17 minutes

Qwen scored perfectly on reasoning, instruction following, and tool use tests. More strikingly, on a task where Ornith needed 18,000 steps and still barely finished, Qwen produced correct, passing code in just 4,000 steps and 2 minutes. It also solved practical logic puzzles and the cybersecurity CTF challenge without issue.

Qwen's only weakness appeared in the context retrieval test, where it missed 1 out of 5 "needle" items hidden across nested documents, scoring 80% — while Ornith found all of them. This highlights an advantage of the MoE architecture in context compression: because different experts can specialize in processing different types of information segments, MoE models tend to perform better at locating scattered information within long documents.

Architecture Differences Define Use Cases

The performance gap between these two models comes down to a fundamental architectural trade-off:

Qwen 2.5 27B is a standard dense Transformer using GQA (Grouped Query Attention). GQA is an attention optimization proposed by Google in 2023 that sits between standard Multi-Head Attention (MHA) and Multi-Query Attention (MQA). In standard MHA, each attention head has its own Key and Value projections, requiring a large KV cache in memory. MQA has all heads share a single set of KV projections to reduce memory usage at the cost of model quality. GQA's compromise is to divide query heads into groups, each sharing one set of KV projections — typically reducing KV cache VRAM by 4–8× while maintaining quality close to MHA. This is a key technical reason why Qwen 27B runs efficiently within 16GB of VRAM. Every inference pass computes all 27 billion parameters, which is slower but logically consistent — making it well-suited for production environments that require stable reasoning.

Ornith 35B uses a 3:1 Mixture-of-Experts (MoE) architecture with 35 billion total parameters, but only 3 billion are activated per inference. The core idea behind MoE is to split the model into multiple "expert" sub-networks, with a gating network dynamically selecting a small number of experts for each inference step. This sparse activation design allows the total parameter count to be very large (meaning greater knowledge capacity) while keeping the computation per forward pass far smaller than an equivalently-sized dense model. Google's Switch Transformer and Mistral AI's Mixtral are landmark works for MoE in large language models. The challenge with MoE, however, is that improper expert routing can cause load imbalance, and switching between different expert combinations across reasoning steps can cause logical discontinuities. This is precisely why Ornith is 28% faster to generate and processes prompts at 3.5× Qwen's rate, yet tends to diverge on complex reasoning chains — strict token limits are needed to prevent runaway generation.

Hardware Configuration Recommendations by VRAM

Before diving into specific recommendations, it helps to understand quantization. Quantization compresses model weights from high-precision floating point (e.g., FP16/BF16 at 2 bytes per parameter) to low-precision integers (e.g., INT4 at 0.5 bytes, INT2 at 0.25 bytes). Common quantization formats include GPTQ, AWQ, and GGUF. Q4 (4-bit) quantization reduces VRAM usage by roughly 75% with a typical quality loss of only 1–3%, making it the most popular approach for local deployment. Q2 (2-bit) quantization further reduces size but causes significant accuracy degradation — especially in MoE models, where aggressive quantization disrupts gating network routing precision, leading to incorrect expert selection and amplifying reasoning instability.

12GB VRAM (RTX 3060 / 4060 / 5070)

• Ornith 35B requires extreme quantization (Q2), dropping speed to ~22 tokens/sec with a high risk of out-of-memory errors
• Qwen 2.5 27B cannot be fully loaded
• Recommended: Ornith 9B or Qwen 2.5 7B

16GB VRAM (RTX 4070 Ti / 5070 Ti)

• Qwen 2.5 27B Q4 loads fully without CPU offloading
• Ornith 35B requires ~4GB of CPU RAM overflow, impacting speed

24GB+ VRAM (RTX 4090 / 5090)

• Both models run at full speed
• Ornith 35B benefits from setting micro-batch size to 2048 for optimal performance

For developers building agentic systems or advanced applications, Ornith also offers a 397B flagship version that achieves near Claude Opus-level coding capability under the MIT license — but requires 192GB of VRAM (8× H100 or a Mac Studio cluster).

Conclusion: Choose the Right Model for Your Use Case

This comparison has no simple winner. Ornith 35B leads in generation speed, prompt processing, and context retrieval — making it well-suited for rapid prototyping and high-volume code generation workflows. But for complex tasks requiring stable reasoning and multi-step planning, Qwen 2.5 27B's 99% accuracy and shorter total runtime make a strong case for the dense architecture.

Developers should choose based on actual needs: Ornith for fast iteration, Qwen for production stability. And for most local deployment scenarios, VRAM constraints often matter more than model preference — a Qwen that runs fully on a 16GB GPU may be far more practical than an Ornith that requires compromised quantization.