How Fixing Three Bugs Made Qwen3.5-122B Run Reliably on Mac Studio

The Last Mile of Local Large Models

Cloud-based inference has long dominated large model deployment, but a growing number of developers and researchers are turning toward local solutions—data privacy, cost control, and offline availability are the core drivers behind this trend. Recently, a developer shared a real-world experience on Hacker News: by fixing three critical bugs, they successfully got Qwen3.5-122B to run reliably on a Mac Studio and turned it into a genuine daily driver.

What makes this case worth analyzing in depth is that it points directly to the most practical pain point in local large model deployment—there's a significant engineering gap between "getting it running" and "using it reliably every day."

Why Qwen3.5-122B and Mac Studio?

Model Choice: The "Sweet Spot" of 122B Parameters

The Qwen (Tongyi Qianwen) series is Alibaba's open-source large language model family, which performs excellently in Chinese and multilingual scenarios and has become one of the mainstream choices in the open-source community. At around 122 billion parameters, it falls into the medium-to-large scale category—its inference quality is noticeably superior to the common 7B and 14B models, while being far more hardware-friendly than the ultra-large models with hundreds of billions of parameters.

For users pursuing high-quality local inference, 122B represents an ideal balance—powerful enough to handle complex code generation, long-text understanding, and logical reasoning, yet "light" enough to potentially be supported by workstation-grade hardware.

Mac Studio's Core Advantage: Unified Memory Architecture

The key reason Mac Studio has become a popular platform for local large model deployment lies in Apple's Unified Memory Architecture (UMA). A Mac Studio equipped with an M2 Ultra or M3 Ultra chip can be configured with up to 192GB of unified memory, where the CPU and GPU share the same memory pool—completely bypassing the hard bottleneck that VRAM capacity imposes in traditional PCs.

Understanding this advantage requires some background. In traditional PC architecture, the CPU has its own dedicated system memory (DDR), and the GPU has its own dedicated VRAM (GDDR/HBM). Data transfer between the two must pass through the PCIe bus, which has limited bandwidth and noticeable latency. Apple's UMA combines the memory pools of the CPU, GPU, and Neural Engine into one, packaged within the same SoC package, and the M3 Ultra achieves memory bandwidth of up to 800GB/s. This is hugely significant for large model inference: the biggest bottleneck during inference is often not the computation itself, but the memory bandwidth required to move model weights from memory to the compute units. UMA not only eliminates the VRAM capacity bottleneck but also dramatically improves data transfer efficiency for each forward pass—this is the fundamental reason why Mac Studio often performs more smoothly than a PC with a single consumer-grade GPU on models of comparable parameter scale.

A quantized 122B model still requires tens of GB of space to load its weights, and Mac Studio's large-capacity unified memory happens to meet this requirement precisely, making it possible to run ultra-large models on a single machine.

Three Critical Bugs: From "It Runs" to "It's Usable"

The original share didn't list the complete technical details of each bug, but based on common experience in local large model deployment, such issues typically cluster around the following three levels.

Memory Management and Loading Stability

When running large models on the Mac platform, one of the most common issues is abnormal memory allocation and release. When model weights approach the memory ceiling, any memory leak or unreasonable caching strategy can cause system stuttering, process crashes, or even trigger macOS's memory pressure protection mechanisms.

Particularly critical here is the management of the KV Cache (Key-Value Cache). The KV Cache is a core optimization mechanism for Transformer inference: during autoregressive generation, each time a new token is generated, the model needs to attend to the attention key-value pairs of all previous tokens. If the K and V matrices of previous tokens were recomputed at every step, the computational cost would grow quadratically with sequence length; the KV Cache stores the already-computed historical key-value matrices in memory, appending only the new token's computation at each step, reducing complexity to linear. However, the KV Cache itself consumes substantial memory—for a 122B model handling long contexts, the KV Cache can occupy several GB or even tens of GB. In local deployment scenarios where memory is near its ceiling, the KV Cache's allocation strategy, reclamation timing, and cache limit settings directly determine whether the model can run stably for long periods without an OOM (Out-of-Memory) crash. Fixing such bugs often means optimizing the model loading process or the KV Cache management strategy during inference, making prolonged continuous use possible.

Inference Performance and Output Consistency

The core requirement of a "daily driver" is stability and response speed. A model occasionally producing results is a completely different user experience from one that consistently, quickly, and reliably responds to everyday queries.

The technical core of this level of problem is the adaptation between Metal (Apple's GPU compute framework) and the inference engine. Metal is similar to CUDA in the NVIDIA ecosystem—it's the low-level GPU compute interface Apple designed for its hardware. Currently, the two most mainstream local inference engines on Mac are llama.cpp and MLX: llama.cpp offloads core operators like matrix multiplication to the GPU via its Metal backend, but since it was originally designed for the CUDA ecosystem, its Metal adaptation layer suffers from incomplete operator coverage and some operations falling back to CPU execution. MLX, on the other hand, is Apple's officially released machine learning framework, specifically optimized for Apple Silicon, making fuller use of unified memory, though its ecosystem is relatively less mature. The quality of adaptation between the inference framework and the Metal backend directly affects token generation speed (token/s) and GPU utilization. After fixing related bugs, both token generation speed and output consistency can often see significant improvement.

Quantization Implementation and Precision Assurance

Running a 122B model within limited memory makes quantization almost mandatory. Quantization is the technique of compressing model weights from high-precision floating-point numbers (such as FP32 or BF16) to low-bit integer representations: take the common Q4 quantization as an example—where each parameter originally required 16 bits of storage, only 4 bits are needed after quantization, theoretically compressing the model size to about 1/4 of the original. A 122B parameter model stored at BF16 precision requires about 244GB, and after Q4 quantization it becomes roughly 61–70GB, fitting neatly into 192GB of unified memory with room to spare for inference buffering.

However, quantization is not without cost. Improper quantization implementations can introduce numerical overflow or excessive precision loss, leading to degraded output quality or even garbled text. Mainstream quantization schemes (such as GPTQ, AWQ, and GGML's Q4_K_M) each strike different trade-offs between compression ratio and precision loss. Choosing the appropriate quantization granularity (per-layer vs. per-group) and calibration dataset is key to ensuring quantization quality. Fixing quantization-related bugs means preserving the model's inference capability to the greatest extent while compressing its size—this is the most critical technical dividing line between "usable" and "good to use."

The Practical Value of Local Deployment

Data Privacy and Full Autonomy

Running a powerful 122B-class model entirely locally means all data stays on the user's device. For developers, enterprises, and researchers handling sensitive information, this is a core value that cloud APIs cannot replace: no worries about data being uploaded, no long-term per-token billing to bear, and full availability even in offline environments.

Rethinking the Cost Structure

While the one-time acquisition cost of a high-spec Mac Studio is considerable, for users who frequently use large models, the total cost of ownership (TCO) may be more advantageous compared to the continuously accumulating cost of cloud API calls. Estimating based on the typical usage of a GPT-4-class API, a heavy user's monthly API expenses could range from hundreds to thousands of dollars; whereas the hardware investment in an M3 Ultra Mac Studio can pay for itself within 1–2 years. This "buy-it-once" model of private compute power is attracting a growing number of heavy users to make the switch.

Lessons for the Open-Source Community

The most far-reaching significance of this case perhaps lies not in which three specific bugs were fixed, but in how clearly it outlines an emerging trend: open-source large models + workstation-grade hardware + community-driven engineering refinement are making high-quality local AI truly within reach.

In the past, running models with tens of billions of parameters or more was the exclusive privilege of data centers; today, a desktop workstation paired with a continuously iterating community inference toolchain can support smooth daily use. Behind this lies the continuous leap in open-source model quality, the increasing maturity of quantization technology (from the early days when only crude Q8 compression was achievable, to schemes like Q4_K_M today that keep quality loss within a very small range), and the patient efforts of countless practitioners—like this developer—in tackling every detailed bug.

Conclusion

From "getting it running" to becoming a "daily driver," the gap between the two is precisely these seemingly trivial yet critical engineering details—fine-grained KV Cache management, deep Metal backend adaptation, and repeated calibration of quantization algorithms. The successful deployment of Qwen3.5-122B on Mac Studio is a real microcosm of the maturing local large model ecosystem. As hardware performance continues to improve and open-source toolchains keep advancing, an era of personal, private, and powerful local AI is accelerating toward us.

Note: This article is based on a brief share on Hacker News. Some technical details are reasonable inferences drawn from general experience in local large model deployment. For specific implementations, please refer to the complete explanation published by the original author.