Open-Source Saluki 27B: A 7.9GB On-Device Agent Flagship That Retains 96% Performance

Squeezing 27B into 7.9GB: An Engineering Feat in Extreme Compression

The team at Underdog AI recently released and fully open-sourced Saluki 27B 1.0, an on-device large language model. Built on Alibaba's open-source Qwen3 27B, it uses quantization and distillation techniques to compress the original 54GB full-precision model down to just 7.89GB — roughly one-seventh of the original size.

What does that number actually mean? Until now, 27B-class models typically required professional-grade GPUs or even multi-GPU setups to run smoothly. At 7.9GB, a consumer-grade GPU or even an ordinary laptop can deliver top-tier model intelligence. The entire series is open-sourced under the Apache 2.0 license with no commercial restrictions — a meaningful signal for developers looking to deploy large models on local devices.

What's truly surprising isn't the compression itself — it's the post-compression performance. Across nine comprehensive authoritative benchmarks, Saluki retains an average of 96% of the original model's performance, shattering the long-held assumption that quantization inevitably causes a catastrophic drop in capability.

Quantization refers to compressing model weights from high-precision floating-point numbers (such as FP32 or BF16) into low-bit integers (such as INT8, INT4, or lower), dramatically reducing storage and VRAM requirements. In Saluki's case, compressing from 54GB to 7.9GB roughly corresponds to reducing each weight from 16-bit to approximately 2–3-bit representation. Distillation takes a different approach: a large model (the teacher) trains a smaller model (the student) by having the student mimic the teacher's output distribution, achieving similar behavior with far fewer parameters. When used together, quantization handles "shrinking the size" while distillation handles "repairing capability loss caused by compression" — the two are complementary. The 96% retention rate is rare precisely because the distillation step can partially compensate for the precision degradation introduced by aggressive quantization, rather than relying solely on the quantization algorithm itself.

Tool Calling: The Unexpected Result Where a Smaller Model Outperforms the Larger One

For modern AI agents, function calling is the core capability that determines real-world usability. Underdog AI invested heavily in specialized tuning for this dimension — with results that are nothing short of dramatic.

On the Underdog Bench, a 120-item function-calling test suite built on the Berkeley BFCL V4 leaderboard, Saluki 27B scored 88 points, directly surpassing the 54GB full-size original's 84 points. According to the team, the smaller footprint actually suppresses the model's tendency toward verbose, rambling outputs, making parameter extraction and format alignment more precise and concise.

This advantage is even more pronounced on concurrent tasks. On the official BFCL 100-item parallel tool-calling test, Saluki scored 42 points, far exceeding the original's 35 points — roughly a 20% improvement in success rate.

Whether handling cross-function coordination for flight and hotel bookings or running concurrent queries across multiple databases, Saluki reportedly achieves zero missed calls and zero parameter confusion. This "orchestration hub" positioning in multi-tool coordination scenarios is critical for on-device agent deployment — after all, a locally running agent needs to be stable and reliable, not just occasionally impressive.

BFCL (Berkeley Function Calling Leaderboard) is a specialized benchmark released by UC Berkeley that evaluates large language models' accuracy in structured function-calling scenarios. It covers multiple sub-dimensions including single function calls, nested calls, parallel calls, and multi-turn dialogues, and is currently one of the most authoritative benchmarks for assessing agent tool-use capabilities. The "parallel function calling" test mentioned here is especially relevant in practice: it requires the model to generate multiple independent function call instructions within a single response and correctly aggregate results when they return — placing extremely high demands on parameter extraction precision and format consistency. This is also a primary failure mode for many large models in real-world agent deployments, manifesting as missed tool calls or parameter confusion.

Code & Instruction Following: Hard Validation of Engineering Capability

Beyond tool calling, Saluki's performance in real engineering scenarios is equally important. On the widely respected SWE-Bench Verified benchmark — which presents 50 complex issues drawn from real open-source projects — Saluki successfully resolved 30 of them, compared to 33 for the full-size original, retaining roughly 91% of real-world bug-fixing capability. According to the team, the model can navigate project codebase context and automatically generate standard patches, making a "fully autonomous, single-machine code assistant" a practical reality.

On strict instruction following, Saluki actually surpasses the original. It scored 93.5 on IFEval (vs. 91.5 for the original) and 72.7 on IFBench (vs. 71.0 for the original). Whether strictly enforcing pure JSON output or precisely controlling word count and paragraph constraints, the model executes with notable accuracy.

On reasoning, the officially reported numbers are refreshingly honest: MBPP+ (code generation) scored 78.0, multi-step causal reasoning scored 67.5, and the model scored 80.0 on the AIME math competition benchmark, retaining roughly 82–85% of the original's mathematical reasoning capability. Pure reasoning is clearly where compression losses concentrate — but for a 7.9GB model, these retention rates are still remarkably solid.

SWE-Bench Verified is one of the closest-to-reality benchmarks for evaluating AI software engineering capability. It draws issues with known, verified patches from real open-source GitHub projects (such as Django, Flask, and NumPy), requiring the model to automatically generate code patches that pass the corresponding unit tests given the repository context. The "Verified" subset has been human-filtered to remove samples with ambiguous descriptions or unreliable tests, making results more credible. Resolving 30 out of 50 issues requires the model to perform code localization, cross-file comprehension, patch generation, and format compliance — not simple code completion. This is why the metric reflects real engineering value far better than traditional benchmarks like HumanEval.

Hot-Swap Architecture and Deployment Experience

Saluki's architectural design reflects a careful approach to local resource management. The 7.89GB main model file is dedicated entirely to text and code logic, preserving precious local VRAM for a 32K context window. When image processing is needed, developers can simply download an additional ~0.9GB visual projection plugin (M-Projector) to seamlessly enable multimodal capability.

This "hot-swap" design ensures every megabyte of hardware resources is used intentionally — if you don't need vision capability, you don't pay the VRAM cost for it.

For deployment, the model uses the standard GGUF format and can be launched with a single command using unmodified llama.cpp, with native support for Qwen3's chat template. It also supports flexible switching between thinking modes: extended chain-of-thought reasoning is enabled by default for complex tasks, while it can be disabled on a per-request basis during high-frequency tool calls for millisecond-level response times.

GGUF (GPT-Generated Unified Format) is a model file format defined by the llama.cpp project, optimized for local inference. It packages model weights, quantization metadata, and chat templates into a single file that can be loaded directly by llama.cpp without additional dependency libraries. Compared to the earlier GGML format, GGUF offers stronger forward compatibility and cross-platform support, and has become the de facto standard for distributing on-device open-source models. For everyday users, GGUF means: download one file, run a single terminal command, and the quantized model runs on a CPU or consumer GPU — no complex Python environment setup or CUDA driver configuration required, dramatically lowering the barrier to local deployment.

A Victory for Intelligence Density

Saluki 27B's true significance isn't any single benchmark score — it's the proof that "intelligence density" can be substantially increased through engineering. Under the Apache 2.0 commercial-friendly license, with only ~7.9GB of memory overhead, it brings 27B-class reasoning capability to ordinary devices, opening a door toward democratized on-device autonomous agents.

Of course, as performance data published from a single source, these benchmark results still await broader community replication and verification — especially the counterintuitive claim that tool calling "outperforms the full-size original," which merits independent third-party cross-validation. But the direction itself is undeniably exciting: when top-tier models are no longer locked behind hardware barriers, local, privacy-friendly AI applications gain a far larger canvas to work with.