192GB Unified Memory Framework Desktop: Running Massive AI Models Locally

When Hardware Meets the Right Models at the Right Time

It looks nearly identical to the previous Framework desktop from the outside, but the internals are a completely different story. The new machine is powered by AMD's Gorgon Halo processor — the Ryzen AI Max+ Pro 495 — and the most significant upgrade is the jump from 128GB to 192GB of unified memory, a full 50% increase in available RAM.

The timing couldn't be better. This lands right alongside the release of next-generation open-weight models like DeepSeek^Ad V4.1 Flash and MIMO V2.6 Flash, and 192GB happens to be the sweet spot for running these large models locally. Among models of their size class, they rank among the strongest open-weight options available today for coding and agent tasks.

On the hardware spec side, Gorgon Halo retains 16 Zen 5 CPU cores and 40 RDNA 3.5 GPU compute units. GPU clock speed nudges up from 2.9GHz to 3GHz, and faster memory pushes theoretical bandwidth from 256GB/s to roughly 273GB/s — about a 7% improvement. But the real change to the user experience is simply that extra memory.

Half a Trillion Parameters on a Single Machine

The most immediate value of the extra memory is being able to run larger models on a single machine, without relying on SSD-based streaming or building a cluster.

DeepSeek V4.1 Flash is a perfect example. Despite having over 500 billion parameters, it uses architectural innovations to reduce the memory required for long contexts, and introduces a mode called SWA-bounded replay. In this mode, most tokens in a long input prompt only pass through roughly half the model's layers, dramatically speeding up prompt processing. It's an approximation that trades some accuracy for speed — but DeepSeek specifically trained the model to accommodate this behavior.

The best current approach for running this model on Gorgon Halo is using the author's Dwarf Star implementation with a Q2 quantized version, where the main model weights occupy around 163GB. Many people are skeptical of 2-bit quantization, but models like this are actually quite tolerant of it — and modern quantization schemes preserve higher precision for the most critical components like attention projections, so quality loss is limited. The full file set is around 366GB (including Ngram tables), but those tables don't all need to be loaded into memory.

In testing, the author tasked the model with building a browser-based 3D particle simulator using three.js, complete with gravity, collision, and attraction forces. This requires the model to simultaneously understand physics, reason about particle behavior, implement simulation logic, and design an interactive interface. The results were impressive.

SWA-bounded replay is an approximate inference strategy introduced in DeepSeek V4.1 Flash, named after Sliding Window Attention (SWA). In a traditional Transformer, every token in a long input goes through the full computation of every layer — memory and compute scale linearly with context length. The core idea behind SWA-bounded replay is that for most earlier tokens in a long prompt, only the shallow layers (roughly half) need to be traversed, skipping the deeper reasoning layers. Full computation is only performed for tokens within the most recent window. The trade-off is some loss of precision, but DeepSeek specifically trained the model for this "lossy fast-forward" mode, enabling it to dramatically reduce both processing time and peak memory for long prompts while keeping accuracy degradation manageable. For tasks like coding and agentic workflows that frequently involve long contexts, the speed gains are especially significant.

Q2 quantization (2-bit quantization) compresses model weights from their original 16-bit or 32-bit floating-point representation down to roughly 2 bits per parameter on average, reducing model size to about 1/8th the original. Conventional wisdom holds that 2-bit quantization causes severe quality degradation. However, modern mixed quantization schemes (like GGUF's Q2_K_XL) preserve higher bit-widths (4-bit or more) for the most precision-sensitive weights — attention projections, output layers, and so on — while aggressively compressing more redundant expert-layer parameters. MoE (Mixture of Experts) architecture models are also inherently more tolerant of weight compression since only a small number of experts are activated per inference. This makes Q2 quantization perform significantly better in practice on 500B-scale MoE models than intuition might suggest.

Model Options and Performance Numbers

Beyond DeepSeek V4.1 Flash, two other large models are worth exploring.

GLM 5.3 Flash has approximately 320 billion total parameters with 18 billion activated per token, and many consider it one of the best open-weight models released in recent months. It's denser than DeepSeek, so current speeds are better suited for long-running tasks than interactive use. The author also created a custom 2-bit/4-bit mixed quantization version to fit Gorgon Halo, trading speed for higher precision.

MIMO V2.6 Flash, released by Xiaomi, has 309 billion total parameters with 15 billion activated per token. Because it's smaller than DeepSeek, GGUF quantization can keep the expert layers in their native MXFP4 4-bit format without further compression, preserving original precision. Its architecture is relatively conventional, but Xiaomi invested heavily in training data and post-training, which explains its strong performance on coding and agent benchmarks.

On performance: using a fresh 2,000-token prompt as a baseline, DeepSeek V4.1 Flash at Q4 starts at around 360 tokens/s for prompt processing, dropping to approximately 342 at 32K context and still holding around 307 at 64K. MIMO in MXFP4 starts faster at around 477, but degrades more steeply, falling to 354 at 32K. Token generation speed is less sensitive to context growth — DeepSeek holds steady around 14 tokens/s, while MIMO drops gradually from 18 to 16. GLM 5.3 Flash is considerably slower overall, generating around 12 tokens/s at Q2 (dropping to roughly 10 at 64K). The author notes that since implementations of these models in llama.cpp are still relatively new, performance should improve further in the coming weeks.

Running Two Models in Memory Simultaneously

Another compelling use of the extra memory is keeping two models resident in memory at the same time.

The author's setup: use the faster but still capable Qwen 3.8 Flash Next as the primary model, while keeping a more powerful model like DeepSeek V4.1 Flash or GLM 5.3 Flash permanently in memory to consult whenever Qwen gets stuck.

Switching models mid-task inside a coding agent isn't ideal — every switch requires the new model to reprocess the entire conversation, which is slow. The smarter approach is to give the primary model a tool that lets it delegate specific tasks to the slower but more capable model. The author experimented with Adobe's secondopinion extension for pi, which allows Qwen to send a problem description and context to a reviewer model, which then analyzes the situation and provides guidance. Qwen can be explicitly instructed to use this tool, or it can decide on its own when it's stuck.

Running this in practice requires understanding how different inference engines (Dwarf Star, llama.cpp, Halogen, etc.) allocate and pin memory, in order to carve out space for both models.

secondopinion is an extension developed by Adobe for pi (an AI coding assistant environment) that implements a lightweight "model routing" mechanism. It's exposed to the primary model as a tool call: when the primary model hits a reasoning bottleneck, it can package a problem description, key context, and specific questions and send them to a designated "reviewer model." The reviewer independently analyzes the input and returns suggestions or step-by-step guidance, which the primary model uses to continue generating. The key advantage is that the reviewer model doesn't need to reprocess the full conversation history — it only receives the distilled problem description from the primary model, significantly reducing the cost of calling the larger model. The primary model also retains autonomy over when to ask for help, avoiding the inefficiency of routing every request through the dual-model path. This is distinct from a full multi-agent framework — it's more like embedding a "hotline" inside the primary model that can be dialed on demand, well-suited for balancing speed and capability in a single-machine multi-model setup.

A Practical Change to the PCIe Slot

Framework also made a welcome change to the PCIe slot. It's a 4-lane short slot, and the standard design has a closed end — the longer connectors used by GPUs (which have 16-lane connections) physically can't fit. Framework has opened up a notch at the end of the slot, allowing longer connectors to extend beyond the edge. No adapter card required, and no need to cut it yourself and risk damaging the motherboard.

This slot has two typical use cases: adding a discrete GPU for heterogeneous inference, letting the integrated and discrete GPUs share the load (a direction the lucebox team is exploring on Strix Halo); or installing a network card such as the Intel E810, which supports RoCE, enabling RDMA over Converged Ethernet to exchange data between multiple machines with lower latency and higher bandwidth for cluster setups.

RoCE (RDMA over Converged Ethernet) is a protocol that implements Remote Direct Memory Access over standard Ethernet. In a traditional TCP/IP network, data transferred between nodes must go through multiple kernel copies and protocol stack processing, resulting in high latency and significant CPU overhead. RDMA allows a machine's NIC to directly read and write another machine's memory, bypassing the CPU and operating system entirely — latency drops to the microsecond range and bandwidth utilization approaches the physical limit. The Intel E810 supports RoCE v2 and can deliver near-InfiniBand communication performance for multi-machine inference clusters over 25GbE or 100GbE networks, at a fraction of the cost of dedicated InfiniBand solutions. For scenarios where a very large model needs to be sharded across multiple Gorgon Halo machines (tensor parallel or pipeline parallel), low-latency inter-node communication is the key bottleneck determining overall throughput — which is why RoCE support is specifically highlighted when discussing NIC use cases.

Conclusion

Gorgon Halo arrived at almost exactly the same moment as DeepSeek V4.1 Flash and MIMO V2.6 Flash — impeccable timing. The extra memory lets these large models run on a single machine without splitting weights, and allows two capable models to reside in memory simultaneously, so the fast model can call on the slow model for help when it gets stuck. For developers looking to deploy very large open-weight models locally, the 192GB unified memory configuration offers a new path that requires neither a cluster nor SSD-based streaming.