Deploying Qwen3 on AMD AI MAX+ 395: GFX1151 Engine Optimization Deep Dive

A Bilibili creator known as "防暴键盘" (Riot Keyboard) recently published an updated deployment guide for their self-developed open-source inference engine, GFX1151 engine, featuring deep optimizations specifically for running the Qwen3 series of large language models (referred to in the video as Qwen3-Flash-Next) on the AMD AI MAX+ 395 platform. Compared to the previous update, the changes this time are substantial — the author even put together a full slide deck to explain the work. This article breaks down the core technical highlights and benchmark results for readers interested in running very large models locally on mobile or desktop platforms.

MOE Kernel Optimization: Major Prefill Performance Gains

The most significant change in this update is a complete rewrite of the MOE (Mixture of Experts) kernel. The author noted that the previous MOE kernel had a known performance bottleneck: prefill performance was stuck around 1200 PP and couldn't be resolved — even with assistance from models like Kimi. The breakthrough finally came by using GPT-OSS 5.5 (referred to in the original as "OPAS 5.5"), pushing prefill performance from ~1200 up to ~1400.

The author is careful to point out that this improvement is somewhat model-specific: "This model has a rather unique architecture, which is what makes this level of performance achievable." In other words, the same optimization techniques can't be directly applied to other models — the gains here depend on Qwen3's particular structural characteristics, not just kernel-level tuning.

MOE (Mixture of Experts) is a sparsely-activated neural network architecture: the model contains a large number of "expert" sub-networks, but only a small subset is activated during each inference pass. The Qwen3 series uses this architecture, giving it a massive total parameter count (~235B) while keeping the number of actually-activated parameters much lower — striking a balance between speed and output quality. The Prefill stage refers to the process where the model processes the entire input prompt in one pass before generating a response, while the TG (Token Generation) stage refers to generating output tokens one at a time. PP stands for Prefill Performance, typically measured in tokens per second. Due to the sparse expert-routing nature of MOE architectures, the compute and memory access patterns differ significantly from dense models, placing unique demands on vectorization and memory scheduling in the underlying inference engine — which is fundamentally why general-purpose optimizations don't transfer directly.

Two Weight Formats: Unified Support for HGN and GGUF

This update introduces unified support for two weight formats, a point the author repeatedly emphasizes. The root problem is that models at this scale are enormous (~120GB), and the HGN (HuggingFace native format) and GGUF formats are incompatible with each other. Various frameworks like llama.cpp and Hologen Flash Silver each do their own thing. To address this fragmentation, the author decided to support both formats natively within their engine.

Each format involves trade-offs:

• HGN format: The PLE (extra plug-in module) uses FP8 quantization, coming in at ~47.7GB. This is the preferred choice when prioritizing performance and lower VRAM usage.
• GGUF format: Leverages the high-quality quantization work from Unsloth (affectionately called "树懒兄弟" / "Sloth Bros"), bringing the PLE component down to Q4 at ~26.8GB — but the overall file size is larger and consumes roughly 10GB more VRAM.

The author's recommendation is straightforward: use HGN if you want faster prefill and token generation speeds; use GGUF if you prioritize output quality or already have GGUF files on your machine. Re-downloading 100+ GB of model weights just to switch formats is a real pain. One important caveat: GGUF format is not supported on Windows.

GGUF is a model weight file format defined by the llama.cpp project that bundles model structure metadata and quantized weights into a single file, making it easy to distribute cross-platform. A large portion of community-quantized models circulate in this format. The HuggingFace native format (abbreviated HGN here) follows PyTorch's safetensors sharded storage layout, which more closely mirrors training-side data organization and theoretically has better affinity with inference engines. Quantization is the technique of compressing model weights from high-precision floating point (e.g., BF16) to low-bit integers (e.g., Q4, FP8), dramatically reducing storage and VRAM requirements at the cost of a slight potential reduction in generation quality. FP8 is 8-bit floating point quantization with minimal precision loss; Q4 is 4-bit integer quantization with higher compression but slightly lower fidelity. Unsloth is an open-source project focused on high-quality, low-loss quantization, and its GGUF quantization releases are well-regarded in the community for their strong precision retention.

Paged KV Cache and Concurrency: Production-Ready Capabilities

Beyond raw performance, this update also adds two features aimed at practical production use cases:

Paged KV Cache: With paged KV, cache utilization is significantly improved. The model no longer needs to perform a full prefill on every request, making it much more efficient for multi-turn conversations and long-context scenarios.

"Budget" Concurrency: The engine now supports concurrent multi-request processing, with generation speed distributed evenly across all active requests. The author gives a concrete example: if single-request throughput is 40 tokens/sec, then with 4 concurrent requests each gets roughly 10 tokens/sec. The author jokingly calls this "very budget" concurrency. Still, it's suitable for lightweight production environments that need to serve multiple requests simultaneously.

KV Cache (Key-Value Cache) is a core optimization in Transformer inference: when generating each new token, the model needs to reference the attention key-value pairs from all previous tokens. Caching these intermediate results avoids redundant computation. Paged KV Cache borrows the concept of virtual memory paging from operating systems — it divides the KV cache into fixed-size memory blocks allocated on demand, rather than pre-allocating a contiguous maximum-length memory region for each conversation. The benefits: previously computed context in multi-turn dialogues doesn't need to be re-prefilled, VRAM utilization is higher, and more concurrent requests can be served without wasting memory. This technique was first popularized by the vLLM project and has since become standard in production-grade inference serving.

On-Device Benchmarks: Stable Performance at Low Power

The author tested on a GMKtec machine with relatively modest power delivery — averaging around 120W, peaking at 130W for less than 10 seconds. Even under these low-power conditions, token generation speed remained stable, with very gradual and smooth degradation in long-context scenarios.

On Windows, 128K context prefill performance averaged around 1000 PP, with TG performance matching Linux. On Linux, HGN and GGUF prefill speeds were roughly on par; HGN had a very slight edge in TG performance, but with MTP speculative decoding enabled, the difference doesn't translate into a meaningful real-world improvement.

In terms of VRAM usage, the optimized build consumes approximately 94.9GB — a critical figure given that the Windows platform only has 96GB of usable VRAM. The ~1GB headroom that remains is, as the author jokes, just enough to "play a casual game." This also explains why GGUF doesn't fit on Windows: it uses noticeably more VRAM, and 96GB is simply too tight.

Control Panel and Project Acknowledgments

This update also ships with a visual control panel for monitoring memory usage and configuring parameters like thinking intensity and the thinking on/off toggle. The author self-deprecatingly calls it "not particularly useful" and "flashy," but it does offer meaningful convenience for everyday users adjusting inference behavior.

The author also gave special thanks to several inspiring open-source projects: Hologen Flash Silver (described as "where the dream began" — the project that showed the author this level of performance was achievable) and the projects that taught the author how to optimize MOE kernels. This speaks to a broader truth: local LLM inference optimization is a field that relies heavily on community collaboration and knowledge sharing.

Summary

For users interested in running very large models locally on unified-memory platforms like the AMD AI MAX+ 395, this GFX1151 engine update offers practical insights: roughly a 15% prefill performance boost through MOE kernel optimization, unified support for both HGN and GGUF weight formats, and the addition of paged KV cache and concurrency support. It's worth being clear-eyed about the limitations, however — these optimizations are highly dependent on the specific architectural characteristics of models like Qwen3 and do not constitute a general-purpose solution. Whether to choose HGN or GGUF, Linux or Windows, ultimately comes down to your hardware constraints and how you weigh speed versus output quality.