Needle: Fine-Tuning a 26M Ultralight Tool-Calling Model Locally — Full Walkthrough

What Is Needle? A Model with Only 26 Million Parameters

In an era where large model arms races routinely involve tens or hundreds of billions of parameters, a model with just 26 million parameters stands out as something genuinely different. It's called Needle, developed by a team focused on on-device inference engines and edge hardware, with one core capability: Function Calling.

Function Calling is the mechanism that allows large language models to interact with external systems. Traditional LLMs can only generate natural language text, but models with Function Calling capability can interpret user intent and convert natural language into structured API call instructions. This capability was formally introduced by OpenAI in June 2023 and quickly became foundational infrastructure for AI application development. Its core value: the model doesn't execute tasks itself — it acts as an "intent parser," precisely mapping ambiguous natural language to machine-executable function calls, dramatically lowering the barrier to integrating AI with existing software systems.

Needle doesn't try to be an all-purpose chatbot. Its focus is razor-sharp: receive a natural language instruction along with a list of available tools, determine which tool to call, and output the required parameters in structured JSON format. In other words, it does one thing — short function calls — and does it extremely well.

Because of its compact size, Needle has an impressively low barrier to entry. It's designed for local assistive functions on phones, watches, and smart glasses, runs fine on an ordinary laptop, and can even be fine-tuned on a single GPU. That's its key differentiator from mainstream large models.

Fully Offline: Using Ollama Instead of Gemini to Generate Training Data

Notably, Needle's model weights, training code, and data pipeline are all released under Meta's open-source license, giving developers full control over the entire training pipeline.

However, the original setup has one external dependency: the official data generator only supports calling Gemini to generate training data by default.

The key modification in this hands-on walkthrough is bypassing Gemini and using a locally running Ollama model to generate training data instead. Ollama is an open-source local LLM runtime that lets developers deploy and run various open-source LLMs on their own hardware without relying on cloud APIs. It has built-in support for quantized formats like GGUF, significantly reducing memory requirements and enabling consumer-grade hardware to run multi-billion-parameter models. The author used a 48GB RTX 6000 GPU on Ubuntu, but emphasizes that for Needle itself, a GPU isn't actually needed at all. For the local data generation step, the Qwen3 model family (Alibaba's open-source model series, ranging from 0.6B to 235B parameters, with strong performance on tool calling and code generation) was used via Ollama — readers can freely substitute their preferred model.

The modification process is straightforward: clone the official repo, run the installer (press Enter to skip when prompted for API keys or weights), install dependencies like Torch, then modify the data generation script to target a local Ollama model instead of Gemini. This achieves fully offline operation.

Data Generation and Training: How 40,032 Samples Were Created

After modifying the script, run the data generator from the repository root. The model automatically generates various user requests, matches them to the correct tool calls, validates them, and saves them as JSON files in Needle's training format.

The final dataset contains 40,032 Needle-format samples covering three typical tools:

• Get weather
• Set a timer
• Toggle lights

Each tool has roughly 130 expression variants. An additional 40 negative samples — cases where no tool should be called — were included. This is crucial: it teaches the model to correctly return an empty result when no tool call is needed, rather than forcing a spurious call. Each row in the dataset is a user request paired with its corresponding tool call instruction.

These 40,000+ samples were then split into training, validation, and test sets, and used to fine-tune the 26M-parameter Needle model on a local GPU — teaching it to output correct call instructions for these three tools. The training process used standard components including a tokenizer and TensorFlow, with the trained model saved to a checkpoint.

Architecture Deep Dive: An Encoder-Decoder Transformer Design

Despite its small size, Needle's architecture is thoughtfully designed, using the classic Encoder-Decoder Transformer structure.

The encoder-decoder architecture originates from Google's 2017 paper Attention Is All You Need, originally designed for machine translation. The encoder compresses the input sequence into a semantic vector representation; the decoder then autoregressively generates the output sequence based on that representation. Compared to decoder-only architectures like the GPT series, this structure tends to be more efficient for tasks where input and output lengths differ significantly and output format is fixed (such as JSON generation) — making it well-suited for narrow-task models like Needle that focus on structured output.

Encoder: Understanding the Input

The user query first enters an embedding layer (a lookup table) that converts text into vectors the model can process. After scaling, the vectors pass through a stack of 12 encoder layers.

Each layer uses a self-attention mechanism so every word can attend to its context, along with GQA (Grouped Query Attention) for memory-efficient processing. GQA is an efficient variant of standard Multi-Head Attention (MHA), proposed by Google in 2023. In traditional MHA, each attention head has its own Key and Value matrices; GQA has multiple query heads share a single set of Key-Value heads, significantly reducing KV Cache memory consumption while preserving model expressiveness and improving inference speed. It has been widely adopted by models like LLaMA 3 and Mistral, and is especially critical for Needle, which needs to run on memory-constrained edge devices.

The model also uses RoPE (Rotary Position Embeddings) to encode word order. RoPE was proposed by Su Jianlin in 2021 and encodes positional information by applying rotational transformations to Query and Key vectors during attention computation. This gives the model an inherent ability to handle relative positional relationships and enables good length generalization — processing input sequences longer than those seen during training. Residual connections ensure training stability, and the model strips out heavier processing modules to stay lightweight.

Decoder: Generating the Output

The decoder consists of 8 layers and generates output tokens one at a time. It uses masked self-attention (only attending to previously generated content) and cross-attention to refer back to the encoder's understanding of the query.

Finally, the output head — a tied linear layer + softmax — converts the decoded numbers back into actual tokens, reusing the same embedding table. The model ultimately outputs a tool call in JSON format, including the function name and required parameters (such as location). This is Needle's complete end-to-end working mechanism.

Results: A 96.7% F1 Score

After training and saving the fine-tuned checkpoint, the results are impressive. On 30 test samples, the model achieved a 96.7% F1 call score.

F1 score is the harmonic mean of Precision and Recall, and is the most commonly used comprehensive evaluation metric for classification tasks. In a tool-calling context, precision measures how often the model selects the correct tool, while recall measures how successfully the model identifies all cases where a tool should be called. A 96.7% F1 score means the model performs well on both dimensions: selecting the right tool and not missing any tool calls. Notably, the test set includes negative samples (cases where no tool call is needed), making the evaluation closer to real deployment conditions — the model must know both when to call and when not to call.

This figure means the model selects the right tool with high probability and almost always fills in the parameters correctly; when no tool call is needed, it accurately returns an empty result. The author ran several live demos — querying weather in Lahore, toggling lights, asking about the distance to the Moon — all with satisfying results.

During an impromptu live test, the input "Is it raining in Sydney right now?" successfully triggered the correct weather tool, converting the natural language query into a clean, accurate tool call instruction.

The Value Proposition of Small Models

Needle's lesson for the industry is clear: not every task needs a large model. For structured, intent-specific narrow tasks like tool calling, a 26M-parameter model with tens of thousands of carefully generated training samples can achieve 96.7% accuracy on edge devices.

More importantly, this walkthrough demonstrates the viability of fully local operation — from data generation (using Ollama instead of Gemini) to fine-tuning, the entire process can be completed offline on a single consumer or workstation GPU, with no dependency on any cloud API. For developers who prioritize privacy, low latency, and edge device deployment, this is a highly compelling technical path.

As wearables like smart glasses and smartwatches become more widespread, Edge AI — executing AI inference locally on the device rather than sending data to the cloud — is becoming an increasingly important technical direction. The arrival of on-device AI chips like Apple Intelligence and the Qualcomm Snapdragon Elite has given smartphones and IoT devices the capability to run small AI models locally. Gartner predicts that by 2026, over 80% of enterprise AI applications will be deployed in edge or hybrid environments. Specialized, lightweight, locally fine-tunable tool-calling models like Needle may well become critical infrastructure for on-device AI.

Key Takeaways