16GB VRAM Showdown: Ornith 35B vs Qwen 35B — Full Benchmark Comparison

As local LLM deployment becomes increasingly mainstream, more developers want to run high-quality open-source models on limited hardware. Tech blogger Luke (Luke's Dev Lab) recently put two popular 35B-class models — Ornith 35B and Qwen 3.6 35B — through a comprehensive head-to-head test on a machine with just 16GB of VRAM. This article summarizes the results of that 24+ hour benchmark marathon to help you decide which local model is right for you.

Background on Both Models

Ornith is a model from Deep Reinforce AI, built on top of Gemma and Qwen 3.5 via post-training. Post-training refers to further optimizing a base pre-trained model using techniques like supervised fine-tuning (SFT) and reinforcement learning (RLHF/RLAIF). Compared to pre-training from scratch, post-training is far cheaper yet can significantly improve instruction-following, safety, and domain-specific performance. By post-training on top of Gemma and Qwen, Deep Reinforce AI is standing on the shoulders of giants — leveraging the massive compute Google and Alibaba Cloud invested in pre-training, while layering on their own training data and RL strategies. This is the dominant path smaller teams use to compete with big labs in the open-source ecosystem. The official benchmarks on Hugging Face are bold: they compare the 35B version directly against Qwen 3.6 35B-A3B and claim superiority across the board.

To verify these claims, Luke downloaded the Q4KM quantized versions of both models (both in Unsloth's GGUF format) and ran them on llama.cpp. Quantization compresses model weights from high-precision floats (e.g., FP16/BF16) to low-bit integers, trading a small amount of accuracy for significantly reduced VRAM usage and faster inference. Q4KM is a 4-bit mixed-precision quantization scheme in the llama.cpp ecosystem — "K" refers to the K-quant grouped quantization strategy, and "M" stands for Medium, striking a solid balance between precision loss and compression. GGUF (GPT-Generated Unified Format) is the file format defined by the llama.cpp project, bundling model weights, tokenizer, and metadata for easy cross-platform deployment. The Unsloth team has heavily optimized the GGUF quantization pipeline, and their quantized models are widely regarded as high quality. The two quantized versions have minor differences, but since both use Q4KM, the comparison is largely fair.

The test system has 16GB of VRAM plus some system RAM — a configuration Luke specifically chose because it reflects what most regular users actually deploy on. The full test suite took over 24 hours to complete.

Inference Speed: Ornith Edges Ahead

Performance testing focused on token throughput across two phases: prefill (processing the prompt) and decode (generating output).

Results:

• Prefill phase: Ornith approaches 200 tokens/sec, about 20 tokens/sec faster than Qwen.
• Decode phase: Ornith holds steady at 43 tokens/sec regardless of output length; Qwen runs about 10 tokens/sec slower overall.

Ornith has a modest but consistent lead in inference speed. Luke suspects this relates to active parameter count. Qwen 3.6 35B-A3B uses a Mixture of Experts (MoE) architecture — "A3B" means approximately 3B parameters are activated per inference, out of a 35B total. A gating network dynamically selects a few expert sub-networks per forward pass, keeping compute at small-model levels while retaining a large knowledge capacity. Ornith doesn't explicitly state its active parameter count, but if it activates fewer than Qwen's 3B, each token generation requires fewer matrix operations, yielding higher tokens/sec on the same hardware.

Long Context: Both Score 93% on 256K Needle-in-a-Haystack

The memory test used the classic needle-in-a-haystack method, introduced by Greg Kamradt in 2023 and now an industry standard. The idea: fill the model's context window with irrelevant "haystack" text, plant a key piece of information (the "needle") at a specific position, then ask the model to retrieve it. The core challenge is that as context grows longer and the needle approaches the end of a full context window, models tend to "forget" due to attention mechanism limitations. This test filled the context to the full 256K and placed data fragments at four depth positions — 5%, 50%, 75%, and 100% — with three repetitions at each depth.

The results were striking — both models performed identically, each scoring 93%. They passed all three trials at depths up to 75%, with a single failure each at the 100% full-context depth.

For quantized models running locally, maintaining such high retrieval accuracy at 256K context is impressive. Both models are essentially tied on long-context performance.

Agentic Tool Calling: Qwen Edges Ahead

The agency test placed each model inside a simulated company sandbox with a set of callable tools, evaluating whether the model could understand the task, reason through the problem, and chain multiple tools together to achieve a goal.

Both models scored 95, with both failing only the hardest task — #13, which required building a tool chain to convert USD to JPY without a direct exchange tool available.

The difference was in how they failed:

• Ornith: Barely found the right direction, mostly just listing which currency pairs were available.
• Qwen: Identified the correct tool chain but failed to invoke the currency conversion tool at the final step, opting to calculate the conversion manually instead.

Qwen got much closer to the goal, giving it a slight edge in tool-calling capability.

HumanEval Coding: Over-Thinking Is the Key Variable

OpenAI's HumanEval, released in 2021, is a coding benchmark with 164 handwritten Python function completion problems. Correctness is verified by actually executing the code (Pass@k metric).

This round revealed an interesting phenomenon — over-thinking. This stems from the "thinking budget" mechanism introduced in newer reasoning models: the model can perform internal chain-of-thought reasoning before giving a final answer, but a token limit caps that reasoning. Once the budget is exhausted, the model can't produce a complete answer:

• Ornith: 45 questions unanswered, likely due to exhausted thinking budget. Pass rate on answered questions was 97%, but overall score was 70%.
• Qwen: Only 29 questions unanswered, 98% pass rate on answered questions, overall score 80%.

Luke theorizes that since Ornith was trained on both Qwen and Gemma, and Gemma tends to "think more," the Gemma component may have made Ornith prone to spending excessive reasoning steps even on simple problems — burning through the thinking budget and leaving questions unanswered. This round goes to Qwen, though the gap isn't huge.

Coding Challenges: Mixed Results

Sand Physics Simulator

This challenge asked the model to implement a physics simulation from scratch with materials like sand, water, walls, and acid.

• Ornith: Succeeded on the first try. Smooth visuals, fast response, acid correctly eroding other materials — a near-perfect result.
• Qwen: Took significantly longer, even hitting context limits requiring compression (prompt of 30,000 tokens exceeded the 32,700 available), water and acid physics were broken in the first output, and additional prompting was needed before it reached a working state.

Ornith wins this round decisively with its first-pass success.

Dungeon Exploration Game

This was a harder task: implement a dungeon random generation algorithm and raycasting from scratch, without using any existing frameworks. Raycasting is a classic 2.5D rendering technique made famous by Wolfenstein 3D (1992) and DOOM (1993). It casts virtual rays from the player's viewpoint toward each screen column, calculates the intersection distance with walls, and renders wall slices of varying heights to simulate 3D perspective on a 2D map. Implementing this from scratch requires understanding trigonometry, the DDA (Digital Differential Analyzer) algorithm, boundary collision handling, field of view, and more — a high-difficulty test of deep programming reasoning.

Both models struggled:

• Ornith: Initial version had no player movement, minimal raycasting range, and no map. Repeated feedback couldn't fix it until the model proactively output debug logs, allowing Luke to collaborate on pinpointing the issue. After about three more iterations, a solid result was achieved.
• Qwen: Also needed multiple prompts but progressed more smoothly — walls, fog of war, and area connectivity were each added in turn, reaching a playable state in fewer iterations.

Worth noting: Ornith's behavior of proactively outputting debug logs is exactly what experienced programmers do when facing bugs. This round is essentially a draw — both needed extended back-and-forth to finish, and both 35B quantized models ultimately produced a playable version.

Final Verdict

Across all tests, Luke's conclusions are:

• Inference speed: Ornith wins (prefill ~20 tokens/sec faster, decode ~10 tokens/sec faster)
• Long context: Dead tie — both excel at 256K
• Agentic tool calling: Qwen edges ahead
• HumanEval coding: Qwen wins, mainly by avoiding over-thinking
• Sand physics simulator: Ornith wins with a first-pass solution
• Dungeon game: Draw

Overall: Ornith is essentially on par with Qwen. Given that this is Ornith's first version, that's a strong result with plenty of room to grow.

For users with 16GB of VRAM, this comparison delivers an encouraging signal: even on consumer-grade hardware, quantized 35B local models are already capable of handling most coding, tool-calling, and long-context tasks. Ornith's performance also validates the post-training approach on top of Qwen and Gemma as a viable path to building competitive models.

Luke plans to test both models on more coding challenges including a 2D driving game and maze agents, and will also evaluate the smaller Ornith 9B. Stay tuned.

Key Takeaways