Qwen3.8 vs Ornith 1.5: Local Inference Speed and Code Review Benchmark on Mac

On a Mac Studio M4 Max (64GB), I ran a head-to-head comparison of two local models using oMLX 0.7.0: Qwen3.8-27B (with MTP multi-token prediction) and Ornith 1.5 35B-A3B (MoE architecture, ~3B active parameters, paired with DFlash speculative decoding). The conclusion is clear — Ornith 1.5 wins decisively on speed, the two are tied on code review capability, and Qwen3.8 has obvious weaknesses in memory footprint and JSON output stability.

Time to First Token: The Decisive Metric

For code agents and code review scenarios, Time to First Token (TTFT) is the real bottleneck. Measured data shows that with an 800-token input prompt, Qwen3.8-27B's TTFT is 3.46 seconds, while Ornith 1.5 needs only 0.49 seconds — a 7x gap. On output speed, Qwen hits 55.1 tokens/sec versus Ornith's 90.4 tokens/sec, a relatively smaller gap.

In chat scenarios, a 3-second wait may be tolerable. But code review requires sending long prompts consecutively, and the wait for the first token becomes the dominant cost. Testing used aiperf 0.12.0, single-model loading, server restart before each run, concurrency of 1, and averaged over 8 requests to ensure comparable data.

The acceleration strategies of the two models are completely different: Qwen3.8-27B is a dense model where all 27 billion parameters participate in computing every token, relying on an MTP head to predict subsequent tokens and verify them in a single forward pass; Ornith 1.5 uses a MoE architecture where only about 3 billion parameters are activated per token, paired with DFlash, which lets a small draft model propose token blocks that the main model then verifies.

TTFT (Time To First Token) refers to the latency from when a request is issued to when the model returns its first output token, typically dominated by the prompt processing (prefill) stage. Unlike output speed (tokens/sec), which only affects subsequent generation, TTFT determines the user-perceived "response start time." In interactive scenarios like code agents or code review that require repeatedly submitting long-context prompts, each submission processes the long prompt from scratch, so TTFT grows significantly with input length and becomes the experience bottleneck. Output speed differences, by contrast, only become amplified when generating long text. Measuring TTFT typically requires fixing input length, concurrency, and warm-up state, otherwise factors like cache hits and scheduling queues can cause significant variance — which is why this article emphasizes single-model loading and restarting the server each run.

MTP (Multi-Token Prediction) is an acceleration technique that lets a model predict multiple subsequent tokens in a single forward pass, typically by attaching several lightweight prediction heads to the main model, generating several candidate tokens at once and keeping the correct results through parallel verification. It does not change the model's output distribution and is a lossless acceleration. DFlash belongs to the speculative decoding paradigm: a smaller, faster draft model proposes candidate token blocks in batches, and the main model verifies them in a single parallel forward pass, accepting matching parts and rejecting and resampling mismatched ones. The core difference between the two lies in the source of drafts — MTP uses the main model's own prediction heads, while DFlash uses an independent small model, so DFlash's speedup ratio tends to be higher, but it requires loading an additional draft model and bearing the overhead of verification failures. This explains the difference in acceleration gains between the two models.

Improvements and Limitations from the oMLX Upgrade

After upgrading from oMLX 0.6.4 to 0.7.0, Qwen3.8-27B's output speed on code prompts rose from 57-59 tokens/sec to 71.7 tokens/sec, and prose content from 46-48 to 55-56. The official release notes mentioned faster MTP verification and decoding, matching the data. Ornith barely changed, holding at 103-121 tokens/sec for code and 99-102 tokens/sec for prose.

I tried adjusting the MTP draft depth: the default is 4 tokens, raising it to 6 dropped output speed to 53.9, and to 8 dropped it further to 52.4, both below the default 55.1. The adaptive controller already selects the optimal depth, so manual intervention yields no benefit.

Code Review: A Tie

With reasoning mode off, Qwen3.8-27B detected 9 of 20 injected bugs, and Ornith 1.5 detected 8 — a one-bug difference is noise at a sample size of 20. The test method was to take 20 commits from four real repositories, flip one operator on added lines (e.g. < to <=, == to !=, deleting not), and ask the model to review the diff in a single call without tools. Each mutated diff was paired with a clean diff, and a pass required flagging the issue in the mutated version without false-flagging in the clean version.

Model	Reasoning Mode	Detected	Unparseable	Clean-diff False Positives	Median Time
Qwen3.8-27B + MTP	Off	9/20	4/40	0.7	10 sec
Ornith 1.5 + DFlash	Off	8/20	2/40	1.6	3 sec
Qwen3.8-27B + MTP	On	6/20	12/40	0.3	194 sec
Ornith 1.5 + DFlash	On	8/20	13/40	0.0	102 sec

With reasoning mode on, Qwen's detection rate actually dropped to 6/20, and 12 of 40 responses exhausted the 12000-token output limit, making the JSON unparseable. Ornith also suffers from overthinking, with 13/40 responses unparseable, but its detection rate held at 8/20 and it took about half of Qwen's time. A review tool that takes 3 minutes and has a third of its responses unparseable is impractical for any model.

Insufficient Memory: The Two Models Cannot Coexist

Qwen3.8-27B occupies 16.1GB after loading, and Ornith 1.5 occupies 19.8GB. On a 64GB machine this seems ample, but oMLX 0.7.0's memory guard refuses to load both simultaneously — the projected memory of 38.24GB exceeds the 34.51GB dynamic cap, returning a 507 error.

In 0.7.0, when one model is idle oMLX evicts it to load the other, and the logs mark the unload as an immediate abort, losing in-flight requests. I accidentally confirmed this: Qwen ran the review test in reasoning mode for 90 minutes while another tool kept sending requests to Ornith, and oMLX repeatedly loaded and evicted Ornith, eventually evicting Qwen mid-response, with 19 of 40 requests failing. The plan of using two models as second opinions for each other is not viable.

MoE (Mixture of Experts) architecture splits the feed-forward layer into multiple expert sub-networks, routing each token to only a few of them, so "total parameter count" and "active parameters per token" are two different concepts. Ornith 1.5 35B-A3B means roughly 35 billion total parameters with about 3 billion activated per token, with daily compute close to a 3B dense model while retaining a larger knowledge capacity. The cost is that all expert weights still need to reside in memory, so its VRAM/memory footprint (19.8GB) is actually higher than the denser but smaller Qwen3.8-27B (16.1GB) — this is the fundamental reason why, despite vastly different compute, neither can coexist on a 64GB machine. Understanding naming like A3B helps you estimate memory budgets for local deployment.

JSON Truncation Issues and the Cost of Constrained Decoding

Qwen3.8-27B returned {"findings":[] and stopped in 5 of 40 review responses, missing the closing bracket, occurring only on empty responses of five or six tokens. Investigation showed it also occurred with MTP off and presence penalty at 0, happening up to 4 times in 15 runs — the model itself occasionally terminates early.

Constrained decoding can eliminate this problem. oMLX supports it natively via xgrammar, but Homebrew installation requires explicit enabling: brew reinstall jundot/omlx/omlx --with-grammar. After enabling the JSON schema, 0 of 30 responses were unparseable. The cost is that grammar constraints and MTP cannot run simultaneously, so such requests fall back to normal decoding and speed drops to about 25 tokens/sec.

Constrained decoding applies a grammar mask to the logits at each step, allowing only tokens that are legal in the current context to be sampled, thereby ensuring output strictly conforms to target grammars like JSON. oMLX implements this via xgrammar, which eliminates structural errors like missing brackets or quotes at the sampling level. There are two main costs: first, the grammar mask must be recomputed at every step, adding overhead; second, the per-token dependencies under grammar constraints break acceleration paths like MTP and speculative decoding that "predict in batches then verify," since those methods assume predicted tokens can be freely verified in parallel. So enabling the schema causes requests to fall back to normal autoregressive decoding, with a significant speed drop. Whether to enable it requires weighing "output parseability" against "throughput speed."

Open Questions and Selection Advice

Qwen's reasoning-mode data mixes two runs: 21 responses from a morning with concurrent traffic, and 19 failed requests re-run in a quiet afternoon environment. I have not yet re-tested all 40 in a single clean run. Qwen's vision capability was also untested — the Ornith quantized version does not support vision, and if Qwen's vision could work with MTP, it might let me retire the third, smaller model I use for images. The cost of Ornith using JSON schema with DFlash is likewise unknown.

Speed tests also need more requests and prompts of several thousand tokens to validate. I expect the TTFT gap to widen with length, but I only tested one size.

For now, Ornith stays. If you are choosing a local model for Mac, measure TTFT at the actual prompt lengths you send before looking at tokens/sec.

One-line Summary

A Mac head-to-head shows Ornith 1.5 with 7x faster TTFT and tied code review versus Qwen, but the two models cannot coexist.

Paragraph Summary

The author benchmarks Qwen3.8-27B (with MTP) against Ornith 1.5 35B-A3B (MoE plus DFlash) on a Mac Studio M4 Max (64GB) using oMLX 0.7.0. Ornith wins decisively on speed: with an 800-token prompt, TTFT is 0.49s versus 3.46s, a 7x gap, while output speed is 55.1 versus 90.4 tokens/sec, a smaller gap — making TTFT more critical than throughput for code agents. Code review detection rates tie (9/20 vs 8/20 with reasoning off), but enabling reasoning drops Qwen to 6/20 with many unparseable responses, taking about twice Ornith's time. Qwen uses 16.1GB and Ornith 19.8GB, and oMLX's memory guard refuses coexistence, with switching potentially losing requests. Qwen occasionally truncates JSON; constrained decoding fixes it but disables MTP and drops speed to about 25 tokens/sec. The author is currently keeping Ornith.