Running a Local AI Coding Assistant on 8GB VRAM: A Practical Guide to Model Selection

Introduction: The Real-World Dilemma of Local LLMs

In Reddit's local LLM community, a developer raised a very common question: he wanted to deploy a local LLM on his machine and use it with tools like Qwen Code or OpenCode for small-to-medium programming tasks. His hardware setup included an Intel Core Ultra 9 285H processor, an RTX 5070 (8GB VRAM), and 64GB of DDR5 RAM.

Reality, however, was far from ideal: he tested several Qwen-series models, and they either ran painfully slow because they couldn't fit entirely in VRAM, or simply didn't work at all. The smaller models that could fit within 8GB of VRAM often failed at critical steps like tool calls.

This case highlights the most common tension in deploying local AI Agents today: the conflict between the VRAM limitations of consumer-grade GPUs and the high model capability requirements of Agent tasks.

Why 8GB VRAM Is a Major Bottleneck

VRAM Determines the Maximum Model Size You Can Run

When deploying an LLM locally, model weights must be loaded into VRAM to achieve usable inference speeds. A rough estimation formula is: number of parameters × bytes per parameter = required VRAM.

• A 7B parameter model in FP16 (half-precision floating point, 2 bytes per parameter) requires approximately 14GB of VRAM;
• After 4-bit quantization (e.g., Q4_K_M), this can be compressed to roughly 4–5GB;
• But beyond the weights, you also need to reserve space for the KV Cache (context cache) and inference runtime.

Quantization is one of the most critical techniques for local deployment today. The basic principle is mapping model weight parameters—originally stored in FP16 or FP32—to lower-bit representations (such as 4-bit or 8-bit), dramatically reducing storage and VRAM requirements. In the llama.cpp ecosystem, common quantization formats like Q4_K_M and Q5_K_M use naming conventions where "Q4" and "Q5" indicate quantization to 4-bit or 5-bit respectively, "K" stands for K-quants (a block-based mixed-precision quantization scheme that uses higher precision for important layers to minimize quality loss), and "M" indicates a Medium configuration, balancing file size against output quality. Unlike simple uniform quantization, K-quants adaptively allocate precision based on the importance of each layer's weights, achieving better model performance at the same compression ratio.

For 8GB of VRAM, this means you can only comfortably run quantized 7B-class models, with limited context length. Once model weights spill over to system RAM (CPU offloading), inference speed drops off a cliff—and this is exactly the root cause of the "very slow" performance the original poster experienced. The performance gap is easy to understand intuitively: NVIDIA GPU memory bandwidth typically ranges from 300–500 GB/s, while DDR5 system RAM bandwidth is usually only 50–80 GB/s—a 5–8× difference. LLM inference is a classic memory-bandwidth-bound task, requiring a full read of the model weights for every generated token. So once some weights fall into system RAM, token generation speed drops proportionally.

Agent Scenarios Place Extra Demands on Models

Ordinary conversational tasks are relatively forgiving for models, but AI Agents (especially coding Agents) have much stricter requirements:

1. Reliable tool calling capability: Agents need to output function calls in specific formats—any format deviation breaks the entire workflow;
2. Longer context windows: Reading code files and maintaining conversation history consumes a large number of tokens;
3. Strong instruction-following ability: An Agent's multi-step reasoning depends on the model accurately understanding and executing system prompts.

Function Calling / Tool Use is the core capability that distinguishes AI Agents from ordinary conversational AI. The mechanism works as follows: the Agent framework describes a set of available tools via JSON Schema in the system prompt (including function names, parameter types, and descriptions). During inference, the model determines when to call a particular tool and outputs a strictly JSON-formatted call instruction (containing the function name and parameter values). The Agent framework parses this structured output, executes the actual function, and injects the results back into the conversation context for the model to continue reasoning. This workflow demands extremely high structured output capability from the model—it needs to not only understand semantics but also strictly follow JSON syntax and correctly match parameter types and names. Large models (70B and above) typically include extensive tool-calling training data, enabling them to reliably produce correctly formatted call instructions. In contrast, 7B and smaller models, with their limited parameter capacity, tend to struggle with this dual task of "understanding intent while precisely formatting output"—they may omit required parameters, produce incomplete JSON, or cause the Agent framework to fail parsing, breaking the entire workflow.

Small models frequently fail at the "tool calling" step precisely because they lack sufficient instruction-following and structured output capabilities.

Model Recommendations for 8GB VRAM

Prioritize Small Models Optimized for Tool Calling

With only 8GB of VRAM, rather than chasing parameter count, pursue "specialized capability." The following model categories are worth testing first:

• Qwen2.5-Coder-7B-Instruct (quantized): A code-optimized model from Alibaba's Qwen series. The 7B quantized version can just barely fit into 8GB, has good tool calling support, and is the most compatible choice for the Qwen Code ecosystem;
• Qwen2.5-7B-Instruct: More balanced general capabilities with solid function calling support;
• Llama 3.1 8B Instruct (quantized): Demonstrates stable tool calling performance among models of similar size.

It's worth noting that these models outperform other similarly-sized models in tool calling primarily due to differences in training strategy. Taking the Qwen2.5 series as an example, Alibaba specifically introduced extensive tool-calling alignment data during the post-training phase and reinforced the model's reliability in structured output scenarios through techniques like RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization). Meta's Llama 3.1 explicitly mentions specialized optimization for tool use in its training documentation. This means that among models with the same 7–8B parameters, versions that underwent specialized tool-calling training may be far more practical in Agent scenarios than larger models lacking such training.

For these models, Q4_K_M or Q5_K_M quantization formats are recommended, striking a balance between VRAM usage and output quality. Specifically, Q4_K_M typically compresses a 7B model to about 4.1GB, while Q5_K_M comes to about 4.8GB. This leaves approximately 3.9GB and 3.2GB respectively for the KV Cache and runtime within 8GB of VRAM, which directly affects the maximum supported context length.

Set Context Length Wisely

Many users overlook a critical point: context length directly consumes VRAM. If you set the context to 32K, the KV Cache will occupy a massive amount of VRAM, causing the model to spill into system RAM and slow down.

KV Cache is a core optimization mechanism for Transformer inference. During autoregressive generation, the model needs to "attend" to all previous tokens when generating each new token. Recalculating the Key and Value vectors for all historical tokens each time would cause computation to grow quadratically with sequence length. KV Cache solves this by caching the previously computed Key and Value tensors from each attention layer—new tokens only need to compute their own Q/K/V and concatenate with the cache. This drastically reduces redundant computation, but the trade-off is that cache memory usage is proportional to "number of layers × number of attention heads × head dimension × sequence length." For a typical 7B model (32 layers, 32 attention heads, head dimension 128), the KV Cache for 16K context length in FP16 requires approximately 2GB of VRAM; doubling the context to 32K doubles that to about 4GB—nearly unbearable within 8GB of VRAM. This is precisely why context length settings are so critical for users with limited VRAM.

For 8GB of VRAM, the recommendations are:

• Limit context to 8K–16K;
• Prioritize keeping model weights entirely in VRAM;
• If you truly need longer context, consider enabling KV Cache quantization (e.g., llama.cpp's --cache-type-k q8_0 --cache-type-v q8_0 parameters, which quantize the KV Cache from FP16 to 8-bit, reducing cache VRAM usage by roughly half with typically minimal impact on output quality).

Deployment Tools and Optimization Tips

Choose the Right Inference Backend

When running local models, your choice of inference engine significantly affects the experience:

• Ollama: Easiest to get started with—it automatically handles quantization and VRAM allocation, ideal for quick experimentation;
• llama.cpp: Offers fine-grained control over GPU offloading layers (-ngl parameter), perfect for squeezing every last bit out of your VRAM;
• LM Studio: Provides a graphical interface, making it convenient to test different models and quantization combinations.

Although all three tools serve local LLM inference, they operate at different architectural levels. llama.cpp is an open-source project initiated by Georgi Gerganov and serves as the foundational layer of the entire local LLM ecosystem—it implements Transformer inference logic in pure C/C++, supports multiple GPU backends including CUDA, Metal, and Vulkan, and pioneered the widely-used GGUF quantized model format. Ollama is essentially a higher-level wrapper around llama.cpp that bundles model downloading, quantization selection, and API serving into a single command (e.g., ollama run qwen2.5-coder:7b), dramatically lowering the barrier to entry while providing an OpenAI API-compatible local interface that integrates easily with Agent tools like Qwen Code and OpenCode. LM Studio is a desktop application that provides a graphical interface for browsing, downloading, and adjusting inference parameters for models, also relying on llama.cpp under the hood—ideal for users unfamiliar with the command line. Which tool to choose ultimately depends on how much control you need: use llama.cpp for precise tuning, Ollama for quick integration into Agent workflows, and LM Studio for intuitive exploration.

For the original poster's scenario, the recommendation is to start with Ollama, get Qwen2.5-Coder-7B's tool calling working first, then gradually optimize.

Leverage CPU and RAM to Share the Load

The original poster has 64GB of DDR5 RAM—an underappreciated advantage. Using llama.cpp's layer-wise offloading, you can place some model layers on the GPU and others on the CPU. While speed will decrease, combined with the relatively new Intel Core Ultra platform (with NPU), it can still deliver acceptable response times for small-to-medium tasks.

Layer-wise Offloading works in direct correspondence with the Transformer's layered architecture. A 7B model typically has 32 Transformer blocks, and llama.cpp's -ngl (number of GPU layers) parameter lets you specify the first N layers to be placed in GPU VRAM for execution, with the remaining layers computed on the CPU using system RAM. During inference, data passes between layers—GPU layers use CUDA acceleration, while CPU layers use memory bandwidth. This means performance isn't simply "all fast" or "all slow" but follows a gradient—the higher the proportion of GPU layers, the closer overall speed approaches pure GPU inference. For an 8GB VRAM setup running a Q4_K_M 7B model, you can typically place 28–30 layers in the GPU (occupying roughly 3.5–4GB of VRAM), with the remaining 2–4 layers plus the embedding and output layers handled by the CPU, achieving near-pure-GPU speeds without triggering massive spillover.

Additionally, the original poster's Intel Core Ultra 9 285H processor has a built-in NPU (Neural Processing Unit), a dedicated AI accelerator that Intel introduced starting with the Meteor Lake architecture. While NPU support for LLM inference is still in its early stages (primarily through the OpenVINO framework) and throughput is far below that of a discrete GPU, it can serve as a third compute source beyond the GPU and CPU for small-batch inference or specific model formats. As the software ecosystem matures, its value will gradually increase.

The key is finding the sweet spot for GPU layer count: place as many layers as possible in VRAM without triggering spillover.

Conclusion: Manage Expectations, Make Pragmatic Trade-offs

Running a local AI Agent on 8GB of VRAM is fundamentally an engineering problem of "optimizing under constraints." It won't match cloud-based GPT-4 or Claude, but for the clearly defined scope of "small-to-medium tasks," a carefully tuned quantized 7B model is perfectly capable.

Core advice for users in similar situations:

1. Don't blindly chase larger models—prioritize specialized small models with strong tool calling capabilities;
2. Control context length to ensure weights fit entirely in VRAM;
3. Make good use of ample system RAM for layer-wise loading;
4. Lower expectations—local models are best suited for privacy-sensitive, offline, or simple repetitive tasks.

As model compression techniques and small model capabilities continue to advance, the experience of running Agents on consumer hardware will only improve. Notably, recent technology trends are addressing this challenge from multiple angles: extreme compression methods like 1-bit quantization (e.g., BitNet) are making progress in academia; inference acceleration techniques like Speculative Decoding can boost generation speed without increasing VRAM usage; and GPU manufacturers like NVIDIA and AMD are gradually increasing VRAM capacity in their consumer product lines. In the foreseeable future, the constraints faced by 8GB VRAM today will likely be substantially alleviated through the co-evolution of hardware and software.