Testing the 29MB Neural 3 Model: Can On-Device Tool Calling Challenge Large Models?

Taking a phrase like "turn off the porch light and set the oven to 400 degrees" and converting it directly into two clean function calls — that's the problem Neural 3 (also known as Needle) is designed to solve. It requires no internet connection and can run on a phone, smartwatch, or even a small development board. The full version is just 29MB, with a minimum viable version around 8MB. This model, released by Cactus Compute — a YC-backed startup based in San Francisco — sparked heated debate among overseas reviewers claiming that "small models are crushing large ones." This article breaks down the technology, real-world performance, and practical boundaries based on video benchmarks.

Structured Output: Betting Against JSON Parsing

Neural 3 and another model released around the same time — Jav (from TypeSafe AI) — are betting on the same premise: most software decisions don't need a full paragraph of text. They just need a choice — which function to call, and which parameters to pass.

Traditional language models generate answers token by token. You have to extract JSON from the output, validate it, and sometimes retry because the model hallucinated a field. Models like Neural skip all of that and produce only structured output that code can execute directly.

The key enabler is grammar-constrained decoding. Neural compiles your function schema into a byte-level grammar, and every token generated must conform to it — so the JSON is always parseable. But there's an important distinction: a structurally valid output is not the same as a correct answer. If you ask about something no tool can handle (say, "the price of gold"), it returns an empty list rather than fabricating an answer.

Architecturally, Cactus calls this a "simple attention network." In a traditional Transformer, the feed-forward layer holds most of the parameters. Cactus removes it and replaces it with a headband MLP — roughly 25,600 parameters per layer instead of 4.7 million. Most of the knowledge is stored in engram tables, using hash lookups for short 2–3 word fragments. Between 70.8 million and 121 million parameters live in these tables; the model reads records rather than performing multiplications. The result is that the 121M-parameter full model has a compute footprint closer to a 50M-parameter model.

Grammar-constrained decoding is a technique that forces a model to produce outputs conforming to a specific format during inference. The target format (such as a JSON schema) is precompiled into a finite state machine or context-free grammar. Before each token is sampled, the engine masks out all tokens that would be illegal at the current state, allowing the model to only sample from valid candidates. This means the model simply cannot "write" content that violates the schema — brackets won't mismatch, field names won't appear out of thin air, and enum values won't go out of bounds. This approach was popularized by open-source inference frameworks like llama.cpp and later systematized by libraries like Outlines and Guidance. Neural 3 compiles schemas down to the byte level — finer-grained than tokens — further reducing the search space for invalid paths. The trade-off: every new schema requires recompilation, and complex nested schemas can introduce additional latency from the constraint computation itself.

Intelligence Slider: One File from Smartwatch to Laptop

Neural 3's most interesting design feature is the "intelligence slider." The model has 20 layers total, and depths from 2 to 4 can run as standalone models. Download a single file, and you can carve out a 4-layer version for a smartwatch or keep all 20 layers for a laptop. The 4-layer version has 29 million parameters, and each additional layer adds roughly 1MB of memory.

The training trick isn't complicated: about 80% of training steps train the full model normally, while the remaining 20% randomly select a depth, run only those layers, and have the smaller version mimic the full model's predictions. Layers 0 and 19 are always retained; intermediate layers are filled in by repeatedly splitting the largest gap.

Surprisingly, the full model actually improves as a result — in their tests, it achieves 3.9% lower final validation loss compared to the same model trained without this "ladder training." The 29MB file size is also partly because weights are compressed to approximately 2 bits each. On small development boards, they report decoding speeds of around 400 to 4,000 tokens per second.

This training strategy is conceptually similar to academic work on Early Exit Networks and Any-Depth Networks. The core idea is to make intermediate layers capable of completing tasks on their own, not just serving as stepping stones to the final layer. Cactus's approach of "having smaller versions mimic the full model's predictions" is a variant of online knowledge distillation: the full model simultaneously acts as teacher while the truncated sub-model acts as student, trained together in a single forward pass. This shares an intuition with Microsoft's Matryoshka Representation Learning and Google's work on elastic inference: one model file embeds multiple capability tiers, and deployment dynamically selects the truncation point based on available resources — no need to maintain separate sets of weights. The full model improves because intermediate layers are forced to retain information more effectively, which is equivalent to applying additional supervision signals to each layer.

Confidence Scores and Three-Way Routing: The Key to Productization

Every response includes a confidence score from a dedicated calibration head. The engine defaults to a threshold of 0.5 — calls below this score are moved to a field called suppressed_calls, and the main call list returns empty.

Neural's suggested routing pattern is practical: confidence ≥ 0.7 executes directly; scores in the middle range surface the call to the user for confirmation; and if nothing is returned, the app responds with "I can't do that." This three-way split may be the most valuable part of the entire design — it explicitly hands the model's uncertainty back to the product logic.

Getting started requires just one install: pip install cactus-needle. The inference engine downloads from Hugging Face once and is cached locally. After adding the @needle.tool decorator to a function, type hints automatically become parameter types and docstrings become tool descriptions. Pass the tools to the Needle class and call agent.run with a natural language prompt — it picks the function, fills in the parameters, executes your code, and returns results. If you just want JSON without execution, agent.complete returns the function name, arguments, a brief one-line rationale, a confidence score, and even peak memory usage.

Extraction works the same way: define a model with fields, pass in text, and call needle.extract. Feed in a shipping email; get back carrier, tracking number, and delivery date as typed objects. The key point: every field it fills is a span copied directly from the source text, making it "grounded" — but if a value isn't explicitly stated in the text, the model can't produce it. Emails with scattered information are much harder to handle than clean invoices.

Benchmark Numbers Need Context

The headline claim is that "Needle 3 beats models 10x its size on mobile tool calling." But looking at the raw base scores from the ladder training post: the 4-layer model scores only 11.7 on Mobile Action and 21 on Droid Call; the full 20-layer model scores 86 and 47 respectively.

Fine-tuning can push each depth up by 18 to 36 points — even the 2-layer model goes from basically unusable to 56.5. But note that Droid Call only has 200 data rows. One developer who ran their own tests only got 32.2%. Cactus officially responded that Needle is task-specific and should be fine-tuned before going to production. In other words, the base model is a starting point, not a finished assistant.

Fine-tuning itself is lightweight: put data in a JSONL file, with each row containing a query, tool calls, and expected answer — irrelevant examples get empty answer lists. Run needle fine-tune, default 10 epochs, training a LoRA adapter on top of the frozen base. If you have no data, there's a command to generate synthetic samples via OpenRouter — using deepseek^Ad v4 flash by default, which is somewhat ironic since that's exactly the model they claim to outperform after fine-tuning.

After fine-tuning, merge the adapter and export .cat files for any depth and platform. Each platform has a precompiled engine under 1MB, covering macOS, Windows, Linux, Android, iOS, watchOS, and even browser via WebAssembly. Fine-tune once, push the 8-layer version to a Raspberry Pi and the full version to desktop.

Two Approaches: Needle's "Small" vs. Jav's "Parallel"

Jav (from TypeSafe AI, founded by Diego Almeida — the name references Kahneman's "System 1" fast intuitive thinking) takes the opposite approach entirely. It doesn't generate token by token at all; instead, it produces the entire structured answer in a single parallel pass, with response times of 70 to 500 milliseconds. In their Doom demo, Jav reads game state about 10 times per second at a cost of roughly $7/hour — $0.042 per million input tokens with output free.

TypeSafe claims Jav "doesn't hallucinate." But The Register points out this only means it can't produce invalid structures — it can still select the wrong option. Some engineers also argue that much of its speed comes from inference strategy: similar results could be achieved by using a standard model with prefill constrained to a single constant token.

The two models solve different problems. Jav is a large cloud-based model capable of handling more complex decisions and massive option sets (like selecting the correct link from thousands); Needle still performs token-by-token decoding under grammar constraints — its speed comes from being small, not from parallel decoding. Needle is Apache 2.0 open-source, with zero per-call cost and data that never leaves the device. Jav provides probabilities across options; Needle provides a confidence score for each call.

In practice, the two can be combined: Needle on a watch handles commands like "skip the song," while Jav handles genuinely intelligent routing decisions on the backend. If your company is currently paying frontier API prices for high-frequency tasks like "convert simple commands to function calls," that's exactly the cost structure these two models are targeting.

Jav's "single parallel pass to produce structured answers" is mechanistically close to Non-Autoregressive Generation (NAR). Traditional autoregressive models generate tokens serially, each depending on the previous one; non-autoregressive models attempt to predict all output positions in parallel within a single forward pass. This approach has been extensively studied in machine translation (e.g., BERT-style masked prediction) but is hard to generalize to open-domain generation because output length is variable and token dependencies are complex. When output is constrained to a structured, finite set of options, parallel decoding becomes much more feasible — because the candidate space for each field is known and bounded, and the model only needs to independently score each slot. This is the fundamental reason Jav can achieve 70–500ms response times on tasks like "which option to select" — not merely an inference strategy difference. The trade-off is that it cannot handle open-ended reasoning scenarios where tokens need to depend on each other.

Real Boundaries: Don't Be Fooled by the Headlines

Beyond the need for fine-tuning, Needle has a few weaknesses that headlines don't reveal. The playground retains conversation state and continues sessions across consecutive requests — if you send unrelated commands back-to-back (say, adjusting the thermostat, then asking an unrelated question), earlier turns may skew the next call, and you need to reset between independent requests.

Tool design is the primary driver of accuracy: one action per tool, named according to how users actually phrase things, with defaults for every required parameter. If a required parameter has no default and isn't mentioned in the sentence, Needle holds back the call rather than guessing — safer, but it can look like a failure if you're not expecting it. There's also a trigger option (accepting regex) that forces a specific tool to fire on a match, covering phrasing that wouldn't appear in the description. Keep the tool list as short as possible — every additional tool gives the model one more chance to pick the wrong one.

For teams looking to ship on-device AI, Neural 3 represents a pragmatic path: accept that it's a starting point, use your own narrow-domain tools and fine-tuning to develop it into a capable assistant, then use confidence scores to handle edge cases. It's not a replacement for large models — it's 29MB worth of local execution, zero cost, and privacy protection in the right scenario.