Running a 26B LLM on AMD Integrated Graphics: Full GPU Inference and mmap Optimization Guide for Unified Memory Architecture

A Fascinating Experiment

A Reddit user conducted a highly representative experiment on their low-power laptop: can the Gemma 26B-A4B model (MoE architecture, ~4B active parameters) run entirely on an AMD Radeon 840M integrated GPU, leveraging the system's 32GB unified memory?

First, let's understand the MoE (Mixture of Experts) architecture design. MoE doesn't have all parameters participate in computing every token simultaneously. Instead, it sets up multiple parallel "expert" sub-networks within the model, with a lightweight Router/Gate Network dynamically selecting only a few experts for computation based on the input. In Gemma 26B-A4B, "26B" represents the total parameter count of 26 billion, while "A4B" indicates that only about 4 billion parameters are actually Active during each inference pass. The core value of this sparse activation strategy is that the model can possess the knowledge capacity and expressiveness of a large parameter count while keeping actual computation (FLOPs) per inference at small-model levels. This makes MoE architecture the ideal choice for running models that "exceed normal capability ranges" on consumer hardware — you need enough memory to store all 26B parameters, but the GPU only needs the compute power to handle 4B-parameter-level calculations.

The question seems simple, yet it touches on an increasingly important but often misunderstood area of local LLM deployment — inference on APU unified memory architectures. With the proliferation of AMD Ryzen AI series processors and Apple M-series chips, more and more users are attempting to run large models on devices without discrete GPUs, and the concept of "GPU VRAM" under unified memory architecture is fundamentally different from traditional discrete graphics.

In traditional discrete GPU systems, the CPU uses system memory (DDR4/DDR5), while the GPU has its own dedicated high-bandwidth memory (GDDR6/GDDR6X/HBM). When a large model needs to run inference on the GPU, model weights must be copied from system memory to VRAM via the PCIe bus — a process with significant latency and bandwidth bottlenecks. In Unified Memory Architecture (UMA), the CPU and GPU share the same physical memory pool — whether it's Apple M-series chips' LPDDR5/5X or AMD APUs paired with LPDDR5. This design fundamentally eliminates data transfer overhead between CPU and GPU; the GPU can directly access model weights in system memory without the "load to VRAM" step. However, the trade-off is equally clear: LPDDR memory bandwidth (typically 50-100 GB/s) is far lower than dedicated discrete GPU memory (e.g., GDDR6X at up to 1 TB/s, HBM3e at several TB/s), making memory bandwidth the core bottleneck for inference performance on unified memory devices.

Hardware Configuration

• CPU: AMD Ryzen AI 5 340
• iGPU: AMD Radeon 840M (RDNA 3.5 / gfx1152)
• Memory: 32GB unified/shared LPDDR
• OS: Fedora 44 Linux
• Driver/Backend: Mesa RADV + Vulkan

A few key technical components are worth elaborating on here. The AMD Radeon 840M is based on the RDNA 3.5 architecture (internal codename gfx1152), AMD's latest-generation GPU microarchitecture for mobile and integrated graphics. Compared to RDNA 3, RDNA 3.5 features targeted optimizations for power efficiency and AI inference. Although its Compute Unit (CU) count is limited, for MoE models where per-inference computation is relatively small, raw compute power is often not the bottleneck.

On the software stack side, this setup uses a fully open-source graphics driver solution: Mesa is the most mainstream open-source graphics driver framework on Linux, and RADV is its Vulkan driver implementation for AMD GPUs. Vulkan is a low-overhead, cross-platform graphics and compute API maintained by the Khronos Group. It provides finer-grained hardware control than OpenGL and has been adopted by inference frameworks like llama.cpp as a GPU compute backend in recent years. Unlike NVIDIA's CUDA ecosystem, Vulkan is an open standard that enables AMD, Intel, and even mobile GPUs to perform general-purpose computing (GPGPU) through a unified interface — this is critical for large model inference on non-NVIDIA hardware.

Under this configuration, Ollama identifies the 840M as having approximately 16GB of VRAM (16193 MiB), with about 14.6 GiB available for the Vulkan device during model loading.

The Core Contradiction: 31/31 Layers Offloaded to GPU, Yet 14% CPU?

The most puzzling phenomenon in this experiment is this: the ollama ps command reports the model as 14%/86% CPU/GPU, making it appear that a significant portion of computation is still running on the CPU.

But the detailed loading logs clearly state:

load_tensors: offloading output layer to GPU
load_tensors: offloading 29 repeating layers to GPU
load_tensors: offloaded 31/31 layers to GPU

All 31 layers were offloaded to Vulkan (GPU). These two pieces of information seem contradictory but actually reveal a critical detail about how Ollama handles memory accounting under unified memory architecture.

The Dual Accounting Mechanism for Memory Buffers

Two lines from the logs are key to understanding the issue:

CPU_Mapped model buffer size = 13752.09 MiB
Vulkan0 model buffer size    = 11705.71 MiB

In unified memory systems, this "dual" buffer accounting is actually normal behavior. Since CPU and GPU share the same physical memory, when llama.cpp uses mmap (memory mapping) to load model weights, it first maps the weights to a CPU-accessible address space (the CPU_Mapped portion), which is then accessed by the Vulkan backend.

This means CPU_Mapped does not indicate that computation is happening on the CPU. It's more of a memory mapping accounting method. The "14% CPU" shown in ollama ps is very likely derived from these tensor buffers held in CPU-side address space via mmap, rather than actual CPU computational workload.

The mmap Warning: Performance Bottleneck Under Unified Memory Architecture

A critical warning appeared in the 26B model's logs:

tensor overrides to CPU are used with mmap enabled
- consider using --no-mmap for better performance

This message directly points to mmap as the likely source of that 14% CPU figure.

mmap (memory-mapped file) is an operating system mechanism that maps a file on disk directly into a process's virtual address space, allowing the program to access file contents as if accessing memory. In large model inference, mmap is widely used to load GGUF-format model weight files: it allows the operating system to lazily load model data from disk into physical memory on demand, rather than reading the entire model file into memory at once. This is particularly useful in discrete GPU scenarios — when a model is too large to fit entirely in GPU VRAM, mmap allows some weights to remain in system memory (or even fall back to disk swap), with the CPU handling overflow layers and the GPU handling loaded layers, implementing a flexible "partial offloading" strategy.

However, under unified memory architecture, mmap's advantages are significantly diminished. Since CPU and GPU already share the same physical memory, there's no "moving from VRAM to system memory" issue. But the indirection layer introduced by mmap — the OS needing to maintain page table mappings from virtual to physical addresses, handling potential page faults, and routing access through CPU-side address space — can actually introduce additional overhead. Specifically, when model size approaches the available memory limit reported by the Vulkan device, llama.cpp may "spill" some tensors into the CPU_Mapped buffer, maintaining them via mmap. Although these tensors physically reside on the same LPDDR chip as GPU-accessible memory, the CPU-side address space and mmap mechanism may involve additional memory copies or address translation overhead.

How to Disable mmap in Ollama to Improve Inference Performance

The original poster tried looking for an OLLAMA_NO_MMAP=1 environment variable but found it seemingly doesn't exist. In practice, Ollama's mmap control is typically achieved through Modelfile parameters or API request parameters:

• Add to Modelfile: PARAMETER use_mmap false
• Or set in API call options: "use_mmap": false

For unified memory devices, disabling mmap often reduces CPU-side tensor processing overhead, allowing weights to reside directly in memory in a GPU-accessible manner, potentially improving inference performance and bringing ollama ps closer to showing "100% GPU". After disabling mmap, model weights are loaded directly into memory via standard memory allocation (malloc), and the Vulkan backend can directly manage these memory regions, avoiding the virtual memory management layer overhead introduced by mmap.

Performance Comparison with the 12B Model

Here's an interesting detail: when running the smaller gemma4:12b (8.1GB), Ollama explicitly reports 100% GPU. This further supports the hypothesis that the 26B model shows 14% CPU because its ~15GB size approaches the Vulkan device's available ~14.6 GiB limit, triggering mmap's overflow mapping mechanism, while the 12B model fits entirely without needing mmap fallback.

In other words, the ratio between model size and available GPU memory directly determines the memory management strategy, which in turn affects the percentage display. This is also a common "trap" under unified memory architecture: although total physical memory (32GB) far exceeds the model size, the available memory share allocated to the GPU by the OS and Vulkan driver is limited (approximately half of physical memory in this case, around 16GB). Anything exceeding this share gets accounted as CPU_Mapped, triggering "overflow" logic similar to discrete GPU scenarios.

iGPU Inference Performance Results: The Practical Value of 17 tok/s

The benchmark results are quite impressive:

• Prompt eval: 27.08 tokens/sec
• Generation: 17.20 tokens/sec

For an integrated GPU running a 26B (4B active) MoE model, a generation speed of 17 tok/s is fully usable. This is thanks to the A4B architecture — although the total parameters are 26B, only about 4B parameters are activated per inference, dramatically reducing actual computation.

So is this the 840M's actual ceiling? The answer is: there's possibly room for improvement. If that 14% of CPU-mapped tensors does introduce additional overhead, disabling mmap could theoretically squeeze out more performance. However, for iGPUs which are memory-bandwidth-limited devices, the ultimate bottleneck is often not in compute units but in LPDDR memory bandwidth.

To understand this, you need to know the memory access characteristics of LLM text generation (decode phase). During token-by-token generation, for each token produced, the model needs to read nearly all model weights (or for MoE models, the weights of activated experts), but each weight participates in only minimal computation (typically a matrix-vector multiplication with one token's hidden state). This means the decode phase is an extremely memory-bandwidth bound operation — GPU compute units spend most of their time waiting for data to arrive from memory. For the LPDDR5 memory in this example, bandwidth is typically in the 50-90 GB/s range. With 4B active parameters (approximately 2-2.5GB after Q4 quantization), theoretically the model weights can be read approximately 25-40 times per second, corresponding to a theoretical upper limit of 25-40 tok/s. The measured 17 tok/s, considering KV cache read/write, attention computation, and other overhead, is already quite respectable.

Therefore, the potential improvement margin may be limited, and 17 tok/s is very likely near this hardware's practical ceiling.

Local LLM Deployment Recommendations for APU Users

Synthesizing this case study, here are the key takeaways for users running large models on AMD APUs or unified memory devices:

Correctly Interpret Ollama's GPU/CPU Percentage Display

The ollama ps percentages under unified memory architecture cannot be directly equated with "compute workload distribution". As long as the logs show offloaded N/N layers to GPU, it means all transformer layers are executing on the GPU. The percentage more reflects memory buffer accounting attribution rather than actual compute scheduling. In traditional discrete GPU systems, this percentage usually accurately reflects compute load distribution, but under unified memory architecture it can be misleading — users shouldn't assume inference efficiency is severely impacted just because they see "14% CPU".

Prioritize Disabling mmap for Performance Optimization

For large models approaching VRAM limits, disabling memory mapping via PARAMETER use_mmap false typically reduces CPU-side overhead.

Pay Attention to Model Size vs. Available VRAM Ratio

Choosing models that fit entirely within available GPU memory (like the 12B model in this example) avoids the complexity of mmap fallback. When selecting models in practice, reserve 10-15% of VRAM headroom for KV cache and other runtime overhead. For example, with Vulkan reporting 14.6 GiB available, keeping model file size within 12-13 GiB would be a safer choice.

Memory Bandwidth Is the True Bottleneck for iGPU Inference

For iGPU inference, LPDDR memory bandwidth is the core factor determining generation speed. MoE architectures (like A4B) are the ideal choice for running large models on these devices by reducing active parameters. From a hardware purchasing perspective, choosing platforms that support higher-frequency LPDDR5X memory (e.g., 8533MHz vs 6400MHz) may improve inference speed more noticeably than selecting more GPU compute units. Additionally, dual-channel memory configuration is crucial for unified memory architecture — a single-channel configuration directly halves available bandwidth, severely impacting inference performance.

Conclusion

This experiment vividly demonstrates the unique characteristics of local inference under unified memory architecture. Traditionally, we're accustomed to the binary thinking of "discrete GPU VRAM vs. system RAM," but in the era of APUs and unified memory, this boundary is blurring. Understanding Ollama and llama.cpp's memory accounting logic and properly configuring mmap are key to unlocking the potential of this type of hardware.

As unified memory platforms like AMD Ryzen AI and Apple M-series become more widespread, efficiently running large models on these devices will become a skill that more and more developers need to master. And the 17 tok/s Gemma 26B inference result proves that local LLM deployment on integrated graphics already has considerable practical value. It's worth noting that software optimization in this field is still rapidly iterating — llama.cpp and Ollama's support for unified memory architecture continues to improve, and future driver updates and framework optimizations are expected to further unlock the inference potential of this type of hardware.