Local Offline AI Tested: 16 Tasks Expose the Hidden Traps of Small Model Selection

A browser assistant should be able to read invoices, compare two quotes, and find the right button on a page. Ideally, it works offline and responds before the user suspects the app has frozen. That was the starting point for the Lastbrowser team's local AI experiment.

They were looking for a "practical minimum bar": a model small enough to run locally, fast enough to deliver real value, and reliable enough to assist with everyday browser tasks — not to chase leaderboard rankings. But along the way, the team discovered that some of the most disappointing results were partly caused by flaws in their own evaluation methodology.

Lesson One: Mismatched Target Hardware

The original target machine was an aging CPU-only system with an Intel i5-7600 and 16GB of RAM. The team wanted to know: could they find a small local model that remained responsive on hardware like this?

But the actual comparative testing was done on a much more capable Windows workstation — an AMD Ryzen 9 5950X, 32GB RAM, and an Intel Arc A770 GPU (16GB VRAM) — using Ollama and a Vulkan build of Prism/llama.cpp as the inference runtime.

The team admitted: "We didn't run these models on that old i5. Results on the Arc A770 don't prove the same responsiveness on an older CPU-only machine." This exposed the first lesson: write down your target hardware before downloading half a model library, or your experiment quietly becomes a different experiment.

Similarly, the team never established a comparable series of measurements for peak memory, VRAM, or power consumption. The file sizes listed in their tables represent stored model weights only — not total runtime memory footprint. Context storage, buffers, the backend, and draft models all carry their own overhead.

What Does "Good Enough" Mean for a Browser Assistant?

Smooth conversation is only part of the job for this use case. A model might produce delightful explanations yet be unsuitable for extracting invoice data into application fields. It might explain how to click a button but fail to generate the required tool call. It might generate tokens blazingly fast while spending seconds on reasoning the user never sees.

The team set out to test a specific set of behaviors: retaining facts from page content, satisfying multiple constraints simultaneously, distinguishing "information is absent" from "the true value is zero", prioritizing newer sources when sources conflict, treating page content as data rather than instructions, responding accurately to corrections without altering unrelated details, producing usable structured output, selecting the right browser tools with precisely preserved parameters, and remaining useful when real-time web information isn't available.

The distinction between "a helpful response" and "a reliable application component" became the central question of the entire benchmark.

29 Configurations, 16 Tasks, 770 Attempts

The candidate model list expanded in several directions: smaller candidates to probe speed limits; MoE (Mixture of Experts) candidates to explore whether sparse activation could deliver more capability at manageable cost; and compressed large models to see whether smaller weight files could still deliver stronger reasoning.

The current dataset contains 770 recorded attempts across 29 model/configuration entries, but only 16 distinct tasks in the unified test suite. These two numbers describe very different things.

The candidate lineup ranged from the 0.70GB LFM 1.2B all the way to the 24.57GB Nemotron 3 Nano 30B-A3B. The team specifically warned about naming conventions: an "E2B" label doesn't automatically equal the total stored parameter count, and an "A3B" suffix doesn't mean the entire model occupies the same memory as a 3-billion-parameter dense model. Local convenience labels are not authoritative model provenance.

Because different candidates required different runtimes — smaller models through Ollama, Bonsai files and subsequent Gemma/Grug comparisons using a Prism/llama.cpp Vulkan build, and larger offload configurations mixing GPU with system memory — the experiment became "a practical comparison of local deployment setups" rather than a controlled isolation of model architectures.

What is MoE? Mixture of Experts (MoE) is a model architecture that divides the network into multiple "expert" sub-networks, activating only a small subset during each inference pass rather than all parameters. Take "Nemotron 3 Nano 30B-A3B" as an example: 30B refers to approximately 30 billion total parameters, while A3B means only around 3 billion parameters are actually activated per forward pass. This means the model's storage footprint is close to a dense 30B model, but its computational load during inference resembles a 3B model. In theory this balances the knowledge capacity of a large model with the inference speed of a small one — but the real-world benefit depends heavily on hardware support for sparse computation and the quality of the runtime implementation. That's why the article cautions that "the A3B suffix doesn't mean the entire model only occupies the memory of a 3-billion-parameter dense model" — the full weights still need to be loaded, runtime memory usage resembles a large dense model, and only the computation is significantly reduced.

16 Tasks: Traps Designed for the Browser

The test suite was initially split into "standard" and "advanced" categories; the team ultimately abandoned that distinction and exposed every candidate to the same set of meaningful, browser-oriented tasks. You might not have noticed: the actual test prompts are in German — translating this article does not turn the dataset into an English benchmark.

The tasks are deliberately targeted. Consider "summarize a status page without fabricating a recovery date": the supplied facts state that a service went offline due to a power failure, that a spare part would arrive on a specific date, but that the restart time was unknown. The trap is subtle — the delivery date is not a confirmed recovery date, and collapsing the story into "service will resume on date X" means adding a fact that wasn't there.

The invoice extraction task requires precise JSON output (field names, decimal conversions, and boolean types are all part of the contract). A human-readable "unpaid" is a different value from a JSON boolean consumed by a downstream application. Then there's resisting page-level prompt injection — a page contains both the real price of €39 and a malicious instruction trying to override it to zero; correct extraction should still return €39, with the malicious text treated as evidence to examine, not a higher-priority instruction.

The team acknowledged honestly: "We evaluated generated calls, not executed browser interactions. These tests don't prove that real-time pages changed as expected."

The Scoring Bug That Made Gemma Look Much Worse

One of the most valuable findings was a flaw in the evaluation method itself. At one point, Gemma 12B appeared to perform terribly: passing only 4 of 16 tasks, while a smaller candidate looked stronger. Someone challenged that result — and that was exactly the right intervention.

The original strict comparison often equated "application cannot parse directly" with "model answered incorrectly." A JSON response wrapped in Markdown code fences, for instance, might have completely correct values — but a direct JSON parser will reject it because of the fences. The team had compressed two separate questions into a single percentage: is the answer substantively correct? Does it satisfy the precise interface contract?

After re-evaluation, Gemma 12B in non-thinking mode achieved 12/16 substantive passes, while strict passes remained at 4/16. "Neither number should replace the other" — substantive improvement matters to someone reading a chat bubble; a strict failure is still a real defect for a browser application expecting a structured object.

The team also cautioned that their review-assist tooling contains case-specific and model-specific decisions, which both document corrections and introduce risks of post-hoc rationalization and inconsistency. "Revised scores are our current assessment of these stored answers — not proof that evaluation has become foolproof."

What are Markdown code fences? Code fences are the triple-backtick markers (```) placed before and after output content, commonly seen in LLM structured output, e.g., ```json\n{...}\n```. Many language models encounter this format extensively in training data and tend to automatically add fences when generating code or JSON to improve readability. However, for downstream applications that need to parse output directly, standard JSON parsers (like Python's json.loads()) will throw an exception upon encountering non-JSON prefix characters, causing an otherwise correct response to be judged as a failure. This reveals a structural tension between "conversational friendliness" and "application reliability" — the former encourages clean, human-readable formatting, while the latter demands strict adherence to the interface contract with no extra characters. Common solutions include explicitly forbidding fences in the prompt, stripping fences in the application layer as preprocessing, or using an inference framework that supports constrained decoding to directly enforce valid JSON output.

Results: Useful Candidates, Not a Universal Leaderboard

Several patterns worth discussing emerged from the candidate performance.

LFM series: Speed is real, but the contract is hard to satisfy. The smallest, LFM 1.2B, achieved a median generation rate of around 208 tokens/s, but current task scores are only 5/16 substantive and 2/16 strict. LFM 2.6B offers more practical middle-ground value, with the thinking variant reaching 12/16 substantive at around 97 tokens/s.

Granite series: Tiny reached 8/16 substantive and 6/16 strict at around 93 tokens/s; Small hit 9/16 on both metrics but only at around 16 tokens/s. Moving from Tiny to Small bought some strict task wins but wasn't a massive capability leap.

Small MoE models didn't automatically solve the problem. EuroMoE, SmallThinker, Ling mini, and LFM MoE 8B all provided a useful reminder: low active parameter counts don't guarantee application-level reliability. EuroMoE also produced 8 technical errors that had to be reported separately from answer quality failures.

Bonsai 2 27B delivered the most striking result: the non-thinking configuration reached 12/16, and Medium thinking hit 15/16 (both substantive and strict). Interestingly, more reasoning didn't yield a meaningful additional gain — but the sample is too small to call it a universal optimum.

Crashes and Overloads Belong in a Separate Column

The dataset contains 33 technical errors, with the largest cluster from the Qwen 3.5 35B-A3B configuration: 22 errors in 32 attempts. The observed sequence was expert-index server errors followed by repeated connection refusals. Once the server crashed, subsequent questions couldn't get a fair inference test.

Nemotron 3 Nano was operationally more disruptive — after severely overloading the system, two requests timed out and the team stopped that configuration. "Two timeouts are enough to show the tested setup doesn't suit our workstation experience, but not enough to rank the model's reasoning quality."

The key distinction here: operational failures still matter — a model that makes the workstation unusable fails a product requirement. But the team needed to be clear about what failed: the hardware/runtime configuration under test, not some fictional set of answers the model never produced.

What is GPU offloading? GPU offloading refers to the practice of moving some model layers or weights to system RAM (or even disk) when model weights exceed the GPU's VRAM capacity, with the CPU handling the corresponding computation. Inference frameworks like llama.cpp support controlling the number of layers offloaded to the GPU via parameters, with unloaded layers processed by the CPU. This hybrid mode makes it possible to run models far exceeding VRAM capacity on consumer hardware, but the cost is that frequent data transfers between CPU and GPU become a bottleneck, significantly reducing inference speed while dramatically increasing system memory pressure. The crashes and timeouts seen with Qwen 3.5 35B-A3B and Nemotron 3 Nano were very likely caused by system resources being exhausted under offload configurations. This also explains why the article emphasizes "the hardware/runtime configuration failed, not the model itself" — the same model weights might perform stably on a machine with adequate resources.

MTP: The Most Encouraging Speed Experiment

The team first measured Gemma 12B in isolation, then added a companion Multi-Token Prediction (MTP) draft model as a separate configuration. Speculative decoding proposes multiple tokens for the target model to verify in parallel.

The results were impressive: in non-thinking mode, adding MTP pushed the median decode rate from 27.02 tokens/s to 64.94 tokens/s, and median request time dropped from 1.90 seconds to 0.89 seconds — while task scores (12/16 substantive, 4/16 strict) remained unchanged. In thinking mode, request time was compressed from 21.63 seconds down to 7.66 seconds.

The team stayed measured about this: "The speed gains are striking, but they were measured in a small-scale comparison with one attempt per task — not a large-scale acceptance rate and energy consumption study." Stored outputs weren't identical across runs, but this isn't sufficient to conclude MTP itself changed quality — backend behavior, numerical details, and small sample sizes could all be factors. For the product, MTP is a promising acceleration option, but it needs its own regression checks and didn't fix Gemma's output formatting failures.

What is Speculative Decoding? Speculative decoding is a technique for accelerating large language model inference by introducing a lightweight "draft model." The draft model quickly generates several candidate tokens, which the target large model then verifies in parallel — accepting tokens that match its distribution and rejecting the rest with resampling. Because the large model verifies multiple tokens in a single forward pass rather than generating them one at a time, overall throughput improves significantly without theoretically changing the output probability distribution. MTP (Multi-Token Prediction) is a variant of this approach that optimizes draft quality by training the model to simultaneously predict multiple future tokens. The actual speedup from speculative decoding depends strongly on how well the draft model's distribution matches the target model — the closer the match, the higher the acceptance rate and the greater the benefit; if the two diverge significantly, frequent rejections limit real gains. This is why the team emphasized the need for independent regression checks.

The Evaluation Itself Needs Auditing

The embarrassing parts don't end with the original strict scoring. The team's AI coding assistant reported that answer review was complete before the review was actually finished — a subsequent check found 105 unreviewed answers still remaining. "This was a false progress claim by the assistant, not a model benchmark result."

There was also a coding issue: text was read on Windows with an inappropriate default encoding, producing garbled characters and different answer fingerprints. The team fixed the mapping and explicitly used UTF-8. All 770 recorded attempts now have review mappings — but the team emphasized: "Coverage doesn't equal correctness. It doesn't mean there are 770 independent human judgments, and it doesn't mean scoring criteria have no residual ambiguity."

What Needs to Change Before the Next Round

The team laid out a clear set of improvements: freeze scoring criteria first — write out expected facts, output schemas, acceptable variants, and critical failures before running any models; test the scorer itself — validate it against known good and bad examples; maintain comparable conditions — use the same task set and retry strategy, record exact runtime builds and model versions, and separate warm inference from cold starts; evaluate structured output interventions as separate configurations; run real browser workflows — execute tool calls, check page state, and verify expected outcomes; protect machine usability — track peak memory and user-visible responsiveness.

For this German synthetic browser task suite, Bonsai 2 27B on Medium thinking is the strongest practical candidate (15/16). But the team was careful: "It deserves another round of controlled testing, not a universal champion badge. Even 16/16 would only prove success on 16 synthetic examples — not safe autonomous browsing."

The clearest lesson is one of engineering judgment: the model, the runtime, the output contract, and the evaluator all need to be tested. A single percentage can hide a failure at any one of those layers.