Underdog Saluki 27B: A Practical Guide to Tool Calling Under 2-Bit Quantization

When a 27B-parameter model is compressed to 2-bit, can it still reliably call tools? Underdog Saluki 27B offers an answer worth taking seriously. This guide focuses on ISTA's GSQ-RCO quantization scheme, breaking down the GGUF header structure, memory usage, and BFCL tool-calling performance in a native llama.cpp environment to help you turn "small footprint, strong capability" into reality on your local machine.

Why 2-Bit Quantization Deserves Attention

The core tension in model quantization has always been this: the higher the compression rate, the harder precision loss is to control. Traditional 4-bit quantization is already relatively mature, but 2-bit means each weight retains only about 4 discrete values—one misstep can break the model's reasoning chain. This is especially true for tasks like tool calling that require strictly structured output, where malformed formatting is often more fatal than content errors.

What makes Underdog Saluki 27B noteworthy is that after 2-bit quantizing Qwen3.8-27B, it still attempts to preserve the function-calling capability measured by BFCL (Berkeley Function Calling Leaderboard). According to MarkTechPost's report, the scheme is directly compared against ISTA's GSQ-RCO quantization method, with a focus on GGUF header information and memory usage. This means the evaluation dimension is not just "can it run" but "does it run correctly."

For local deployment users, a 27B model at 2-bit could potentially bring memory requirements down to a range affordable by consumer GPUs or even high-memory CPUs—this is its most practical appeal.

Technical Background of the GSQ-RCO Quantization Scheme

GSQ typically refers to Group-wise Scalar Quantization, whose core idea is to partition weights into blocks by channel or group and compute a scale and zero-point independently for each group, thereby reducing the damage outliers cause to overall precision at extremely low bit widths. RCO is commonly understood as modeling residual or compensation terms—that is, retaining a small amount of correction information beyond the main quantization to mitigate the information loss caused by having only about 4 discrete values per weight at 2-bit. ISTA's scheme belongs to the same category as post-training quantization approaches like GPTQ and AWQ, but places greater emphasis on usability at extremely low bit widths. Understanding this helps you judge that 2-bit precision does not degrade uniformly—it depends on whether the grouping granularity and compensation mechanism cover the weights sensitive to structured output.

GGUF Header and Memory Usage Analysis

GGUF is the standard model format in the llama.cpp ecosystem, and its header carries key information such as quantization type, tensor layout, and metadata. Reading the header is equivalent to having the model's "health report" before loading it.

In the context of this guide, the value of the GGUF header lies in two things: first, confirming whether the quantization configuration matches the GSQ-RCO scheme, avoiding downloading the wrong variant; second, estimating actual memory usage. The theoretical compression rate of 2-bit quantization is about one-eighth that of FP16, but actual memory also includes KV cache, context length, and runtime overhead. For a 27B model, even if weights are heavily compressed, the KV cache in long-context scenarios can still become the bottleneck—this is especially evident in tool-calling tasks, since multi-turn function calls quickly drive up context length.

In practice, it's advisable to first run baseline tests with a smaller context window, then gradually scale up, observing the inflection point in the memory curve and output quality.

Memory Composition of KV Cache and Quantized Weights

Actual memory for local deployment = quantized weights + KV cache + runtime overhead. The size of the KV cache is approximately 2 × number of layers × attention head dimension × context length × bytes per element, independent of the weight bit width—so 2-bit compression only shrinks the weight portion. For a 27B model, when context expands from 4K to 32K, the KV cache can grow from hundreds of MB to several GB. After weights have been compressed to the tens-of-GB range, it instead becomes the largest incremental item. This is also why the article emphasizes "start with a small context as a baseline, then gradually scale up." If VRAM is tight, consider KV cache quantization (such as Q8/Q4 cache) or shortening the context, but you need to assess the impact on retaining historical information in multi-turn tool calls.

Key Points from BFCL Tool-Calling Tests

BFCL is a common benchmark for evaluating a model's function-calling capability, covering scenarios such as simple functions, multi-function selection, and parameter filling. The original article explicitly treats BFCL performance as a core evaluation item, providing a comparable yardstick for the practicality of 2-bit quantization.

It must be honestly pointed out that 2-bit quantization typically affects structured output more than natural language generation. Tool calling requires the model to output JSON strictly conforming to a schema, and any token-level deviation can cause parsing failure. Therefore, evaluation should focus on three types of metrics: whether the call is correctly triggered, whether the function name matches, and whether parameters are complete and of the correct type. Looking only at overall accuracy can easily mask systematic degradation at the parameter level.

Another easily overlooked variable is the inference backend. The original article emphasizes testing under "stock llama.cpp" (an unmodified native version), which improves reproducibility—if your environment also uses native llama.cpp, these conclusions will have higher reference value.

Evaluation Structure of the BFCL Benchmark

BFCL (Berkeley Function Calling Leaderboard) is maintained by the Berkeley team, and its evaluation set is tiered by difficulty: Simple tests single-function calls, Multiple tests selecting the correct one among multiple candidate functions, Parallel tests outputting multiple parallel calls at once, Parallel Multiple combines multiple functions with parallelism, and Relevance Detection tests whether the model can correctly abstain when no call is needed. Scoring typically combines AST structural matching with executability checks, so any mismatch in function name, parameter key name, or parameter type is marked wrong. This explains why metrics should be observed by category under 2-bit quantization—Simple may still maintain a high pass rate, while Parallel and parameter-filling tasks often degrade first, and overall accuracy masks this structural difference.

Practical Path for Running in llama.cpp

The whole process can be summarized into a few key actions: obtain the correct GGUF quantized file, verify the header metadata, set context and batch parameters according to hardware conditions, and finally run a smoke test with tool-calling samples.

At the parameter level, it's advisable to prioritize controlling context length and batch size, as these two have the most direct impact on peak memory. Sampling parameters such as temperature should be kept at low values in tool-calling scenarios to reduce format drift. For test order, first use a single-function call to verify the basic chain, then transition to multi-function and multi-turn calls, so you can quickly identify whether the problem lies in model capability or misconfiguration.

For the comparison between GSQ-RCO and Underdog Saluki 27B, the original article provides directional reference, but specific figures should be verified against the original guide to avoid deviations introduced by secondhand retelling.

Applicable Boundaries and Trade-off Recommendations

2-bit quantization is no free lunch. It suits scenarios that are memory-constrained, latency-sensitive, and able to accept some precision loss; if your tool-calling chain demands extremely high parameter accuracy, 4-bit or higher precision remains the safer choice.

A pragmatic strategy is layered deployment: use the 2-bit version for lightweight, high-frequency calling tasks and reserve the high-precision version for critical paths. This way you enjoy the cost advantages of compression without taking risks in core areas. The comparative study of Underdog Saluki 27B and GSQ-RCO is essentially providing data support for this kind of layered decision-making.

One-line Summary

Breaking down Underdog Saluki 27B's 2-bit quantization scheme and key BFCL tool-calling test points under llama.cpp.

Paragraph Summary

This article focuses on Underdog Saluki 27B, a 2-bit quantization of Qwen3.8-27B, and, in conjunction with ISTA's GSQ-RCO scheme, discusses how well tool-calling capability is preserved under extremely low-bit quantization. The article first explains the core tension of 2-bit quantization—compression rate and structured output precision are hard to achieve simultaneously—then analyzes how the GGUF header can be used to verify quantization configuration and estimate memory, pointing out that KV cache is the main bottleneck in 27B long-context scenarios. It then uses BFCL as a yardstick, emphasizing that evaluation should examine call triggering, function name matching, and parameter completeness in detail rather than only overall accuracy, and notes that native llama.cpp testing improves reproducibility. The practical section provides a path for verifying headers, controlling context and batch size, and layered smoke testing, and finally recommends layered deployment of 2-bit and high-precision versions according to task importance.