Local LLM Agent Lagging? Lowering num_gpu Is the Key

An Overlooked Parameter for Local Model Tuning

When deploying large language models (LLMs) locally, many developers encounter a puzzling phenomenon: the model performs well in simple conversations, but once placed inside an Agent framework (agent harness) for multi-step tasks, performance degrades noticeably—or it even crashes outright.

An Agent framework refers to a software architecture that gives large language models autonomous action capabilities. Unlike simple Q&A interactions, Agent frameworks allow models to call external tools (such as search engines, code executors, API endpoints), perform multi-step reasoning, maintain task state, and dynamically adjust next steps based on intermediate results. Typical Agent frameworks include LangChain's AgentExecutor, AutoGPT, CrewAI, and others. In this architecture, each interaction round accumulates previous tool call results, Chain-of-Thought reasoning, and system instructions into the context window, causing token consumption to far exceed that of ordinary conversation scenarios.

Recently, a Reddit user shared their experience, resonating widely across the community. They wrote: "Why didn't anyone tell me sooner that lowering num_gpu could make the model run better in Agent frameworks? I spent months struggling to run models locally, and after lowering num_gpu, my Hermes model finally works properly on my existing hardware."

The Hermes mentioned here is an instruction-tuned open-source LLM series developed by the NousResearch team. It's renowned for excellent Function Calling and tool-use capabilities, making it particularly well-suited for Agent scenarios. Hermes models are typically fine-tuned on base models like Llama and Mistral, enabling them to accurately parse structured tool-calling instructions and generate properly formatted outputs. Because Hermes is designed specifically for Agent scenarios, it utilizes the context window very intensively—which also explains why VRAM management is especially critical for this model.

This seemingly simple discovery actually involves a commonly misunderstood parameter mechanism in local inference engines (especially within the Ollama / llama.cpp ecosystem).

What Does num_gpu Actually Control?

It's Not GPU Count—It's Layer Offloading

First, let's clarify a common misconception: in Ollama and llama.cpp, num_gpu does not mean "how many GPUs to use." Instead, it specifies how many model layers to offload to the GPU for computation. This parameter corresponds to -ngl (number of GPU layers) in llama.cpp.

llama.cpp is an open-source C/C++ inference engine developed by Georgi Gerganov, specifically optimized for Meta's LLaMA series and derivative models. Its core innovation lies in supporting model quantization and CPU/GPU hybrid inference, enabling large language models to run on consumer-grade hardware. Ollama is a higher-level wrapper built on top of llama.cpp, providing a Docker-like model management experience where users can pull, run, and configure models with simple commands. Both share the underlying layer offloading mechanism, which allows users to specify which Transformer layers run on the GPU while the remaining layers fall back to CPU execution.

• When num_gpu is set high (e.g., equal to the model's total number of layers), as many layers as possible are loaded into VRAM for accelerated computation;
• When set lower, some layers remain in CPU and system memory.

Intuitively, putting more layers on the GPU should be faster. So why does lowering it actually make Agent tasks run more smoothly?

The Chain Reaction of VRAM Exhaustion

The key lies in context window length. Agent frameworks are fundamentally different from regular chat: they need to process system prompts, tool definitions, multi-turn conversation history, intermediate reasoning steps, and more. The context easily surpasses thousands or even tens of thousands of tokens.

The longer the context, the more VRAM the KV Cache (Key-Value Cache) consumes. KV Cache is a critical optimization mechanism for Transformer inference. During autoregressive generation, the model needs to perform attention computation over all previous tokens for each new token generated. Without caching, every step would require recomputing the Key and Value matrices for all historical tokens, with computational cost growing quadratically with sequence length. KV Cache stores the already-computed Key and Value vectors from each attention layer, reusing them directly during subsequent generation. Its VRAM usage roughly follows the formula: 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes. For example, a 32-layer, 32-head, 128-dimension model processing 8192 tokens at FP16 precision requires approximately 2GB of VRAM for KV Cache alone. In Agent scenarios, since context length grows dynamically and typically reaches the model's context window limit, KV Cache VRAM consumption becomes very substantial.

If num_gpu is set too high, the model weights already fill up VRAM. Add the inflating KV Cache on top of that, and you get:

1. Out of Memory (OOM): Inference fails outright or crashes;
2. Frequent VRAM-to-RAM swapping: The system is forced to shuttle data back and forth between VRAM and system memory, causing performance to plummet. Even a PCIe 4.0 x16 interface only provides about 32GB/s of bandwidth, while GPU internal VRAM bandwidth typically ranges from 500GB/s to 2TB/s (e.g., the RTX 4090 offers approximately 1TB/s). This means that once frequent swapping occurs, effective bandwidth drops by one to two orders of magnitude, manifesting as inference speed plunging from dozens of tokens per second to just a few or even lower;
3. Degraded output quality: Some engines silently truncate context when VRAM is insufficient, causing the model to "lose memory" and making Agent behavior erratic.

After lowering num_gpu, sufficient VRAM headroom is freed up for KV Cache, which actually stabilizes the entire inference pipeline—this is the fundamental reason why that user's Hermes model "finally works properly."

How to Find the Right num_gpu Value

Balancing VRAM Allocation Between Model Weights and KV Cache

Local inference is fundamentally a VRAM allocation battle between two main contenders: model weights and KV Cache. You need to reserve enough space for both.

Here's a practical tuning approach:

• First, use nvidia-smi or similar tools to confirm your total available VRAM;
• Estimate model weight usage (the quantized GGUF file size serves as a good reference);
• Reserve space for the long-context cache needed in Agent scenarios;
• Start with a lower num_gpu value and gradually increase it until you approach the VRAM limit without overflowing.

Here's a concrete example: Suppose you have an RTX 4070 with 12GB VRAM and want to run a Q4_K_M quantized 13B model (weights approximately 7.5GB) while supporting 8192 tokens of context. KV Cache in this configuration might need about 2-3GB of VRAM, plus roughly 0.5-1GB for engine overhead, bringing total demand to about 11GB. If you offload all 40 layers to the GPU, the weights alone fill up the space, leaving no room for KV Cache. Reducing the layer count to around 30 (approximately 5.6GB of weights) leaves comfortable headroom for caching.

Coordinated Tuning with Quantization and Context Settings

Besides adjusting num_gpu, there are several complementary approaches:

• Choose more aggressive quantization (e.g., Q4_K_M instead of Q8) to shrink weight size and free up more VRAM. The GGUF format supports various quantization schemes, each making different trade-offs between model size, inference speed, and output quality. Take a 70B parameter model as an example: the FP16 version requires about 140GB of storage, Q4_K_M quantization brings it down to about 40GB, and Q8 sits at roughly 70GB. Choosing more aggressive quantization means weights occupy less VRAM, leaving more room for KV Cache;
• Set num_ctx sensibly: Don't blindly max out the context window—use only what you need to avoid unnecessary KV Cache bloat. Many engines pre-allocate KV Cache space for the entire context window by default, even if you won't actually use that much. Reducing num_ctx from 32768 to 8192 might save several gigabytes of VRAM;
• Enable KV Cache quantization (supported by some engines) to further compress cache usage. llama.cpp has recently introduced Q8 and Q4 precision KV Cache quantization options, which can halve or even quarter cache usage with virtually no impact on output quality.

Through the combination of these approaches, even consumer-grade GPUs can support relatively complex Agent workflows.

Takeaways for Local AI Developers

The most thought-provoking aspect of this case is: The pain points of local model deployment often lie not in the model itself, but in insufficient understanding of the underlying parameter mechanisms.

Many developers, like that Reddit user, spend months "battling" with local models, suspecting the model isn't capable enough or that the hardware is insufficient, while overlooking a few critical runtime parameters. Community documentation for parameters like num_gpu is often vague, which exacerbates the misunderstanding.

For developers looking to run Agents locally, here are some recommendations:

1. Understand your inference engine parameters: num_gpu, num_ctx, and quantization level all significantly impact real-world performance;
2. Distinguish between chat scenarios and Agent scenarios: The latter demands far more from context and stability, requiring targeted parameter configuration. Simple chat might only need 1000-2000 tokens of context, while in Agent scenarios the system prompt alone might consume 2000-3000 tokens, plus tool definitions, history, and reasoning steps—8000-16000 tokens is the norm;
3. Leverage community experience: Many "hidden tricks" aren't written in official documentation but are scattered across community discussions on Reddit, Discord, and elsewhere. Subreddits like LocalLLaMA, llama.cpp's GitHub Issues, and various Discord servers have accumulated a wealth of practical tuning experience.

As the local LLM ecosystem continues to mature, mastering these tuning details will be a crucial step in fully unlocking the potential of consumer-grade hardware. Sometimes, the answer to making your model "just work" is hiding in one underestimated parameter.