GPU Prices Going Crazy? Deploy LLMs with Unified Memory + Discrete GPU Heterogeneous Acceleration

GPU Prices Are Out of Control — and So Is the Cost of Local LLM Deployment

With high-end GPU prices spiraling out of control, deploying large language models locally is becoming an increasingly steep barrier for everyday users and small teams. A Bilibili creator, while upgrading his hardware, stumbled onto a new approach: combining idle unified-memory devices with a discrete GPU to run large models through heterogeneous acceleration. Behind this idea is a rapidly evolving open-source direction that's well worth a deep dive.

GPU Prices Are Out of Control — and So Is the Cost of Local LLM Deployment

First, some numbers that will make you wince. NVIDIA launched the RTX 5090 at an official price of $1,999; it landed in China at roughly ¥16,000–17,000. Now, reported international prices have soared to $4,525 (as of September 13), and even at that price it's nearly impossible to find in China.

Even wilder is the card the creator calls his "dream GPU" — the RTX Pro 6000. Early pre-order pricing was under $8,000; by launch it had risen to roughly $8,565 (over ¥50,000). At the time, many reviewers dismissed it as overpriced — similar performance to the 5090, just with three times the VRAM for several times the cost. Then NVIDIA effectively doubled the price to $16,000, with Amazon listings commonly hitting $19,999.

The root cause is VRAM supply. Media analysis points to manufacturers shifting production capacity heavily toward HBM memory, creating a shortage of the GDDR memory used in the Pro 6000 and 5090, driving prices ever higher. For ordinary users, both cards have blown past any reasonable consumer price point.

In this environment, heterogeneous acceleration schemes that squeeze every ounce of use out of whatever hardware you already have are becoming increasingly relevant.

Why Heterogeneous Acceleration Works: Breaking Down the Two Phases of LLM Inference

To understand how heterogeneous acceleration works, you first need to see that LLM inference really only does two things.

Prefill: The "Reading" Phase

Prefill is the pre-loading stage — think of it as the model "reading the question." If you paste in an 8K-token document, the model processes it all in parallel at once, which is essentially a large-scale matrix operation. This phase is compute-hungry, and its output is a set of KV states — what we commonly call the context memory. The creator's analogy is apt: Prefill is like quickly scanning a book full of answers before an open-book exam.

KV Cache (Key-Value Cache) is the key concept for understanding what Prefill produces. When the Transformer architecture processes each token, it generates a Key vector and a Value vector for that token; these pairs are cached and reused in all subsequent generation steps. The essence of the Prefill stage is computing all the KV pairs for every input token at once and writing them into the cache. The larger the cache, the longer the context the model can "remember." This is why long-context scenarios are so VRAM-hungry — a 32K-context KV Cache on a 70B-scale model can occupy several gigabytes of VRAM, and grows linearly with sequence length.

Decode: The "Writing" Phase

Once Prefill is complete, the model enters Decode — generating the response one token at a time, which is the word-by-word text generation you see during a chat. Decode is sequential; weights and KV data must remain resident in memory, so it demands memory capacity and memory bandwidth. The analogy: after scanning the book, writing out the answer one character at a time.

The key insight is: compute-powerful GPUs are suited for Prefill; machines with large memory (unified memory or large VRAM) are suited for Decode. Heterogeneous acceleration is founded on assigning each type of task to the hardware best equipped for it — high-compute GPUs handle dense calculation and long inputs, while large-memory devices carry the weights, context, and token generation — achieving a "1+1 > 2" outcome. The prerequisite is that memory bandwidth on the Decode side must not be too slow, or it becomes the bottleneck.

Real-World Benchmarks: Three Paths and a Six-Device Setup

The open-source project presents three collaboration paths around the Qwen3 27B dense model — the same model, three different hardware combinations:

• Path A (Layer-split assist): An RTX 3060 handles the front layers for GPU compute; an AMD AI Max+ 395 handles the back layers. Under IQ3 quantization with 64K long inputs, Prefill speed jumps from 136.69 tokens/s to 319.1 tokens/s, while also solving the single-machine memory problem.
• Path B (Stage separation): An RTX 3080 does Prefill; the AI Max 395 does Decode. Under Q4 quantization, time-to-first-token drops from 4.825 seconds to 1.073 seconds — just one-quarter of the original.
• Path C (Dual-device collaboration): An RTX 3080 paired with a DGX Spark boosts Prefill speed from 1,111.3 to 1,600 tokens/s — roughly a 44% improvement. The trade-off: Decode speed falls from 33.36 tokens/s to 17.62, because DGX Spark's memory bandwidth is lower than dedicated GPU VRAM.

This data clearly illustrates the trade-offs of heterogeneous acceleration: it can dramatically improve Prefill speed and time-to-first-token, but Decode speed is constrained by memory bandwidth.

IQ3 and Q4 are two different model quantization levels. Quantization is the technique of compressing model weights from 32-bit or 16-bit floating point to lower-bit integers, trading memory footprint for some precision loss. Q4 means 4-bit quantization — currently the most common balance point for local deployment. IQ3 is 3-bit "Importance-aware Quantization," which dynamically allocates precision based on weight importance, maintaining quality close to Q4 at an even lower memory footprint. Qwen3 27B at IQ3 quantization is roughly 11–12 GB; at Q4, roughly 15–16 GB — a difference that directly determines which devices can load the full model weights.

Running DeepSeek V4.1 Flash for $40K — One-Fifth the Traditional Cost

The project's most impressive showcase is a six-device hybrid acceleration setup: four DGX Spark units plus two RTX 6000D cards, enabling local deployment of DeepSeek^Ad V4.1 Flash.

On cost: a single DGX Spark is about $4,700, so four units run roughly $18,700; two RTX 6000D cards cost about $19,000. Total: approximately $38,000, or around ¥270,000 (before taxes, cooling, host RAM, and storage).

That sounds expensive — until you compare it to the traditional approach. The creator notes that to deploy DeepSeek V4.1 Flash at comparable quality via the conventional route, you'd typically need six H20 GPUs plus a Xeon processor and supporting hardware, putting the total close to $200,000. The heterogeneous acceleration setup cuts that cost to roughly one-fifth.

Interestingly, the two RTX 6000D cards together barely cost as much as a single Pro 6000 — yet two Pro 6000s alone couldn't achieve this kind of deployment. That's exactly where the value of collaboration lies.

Compute Rental and Coding Agents: The Project's Extended Ecosystem

Given that DGX Spark (around ¥30,000+) and AMD mini-PCs (¥20,000–30,000) are still significant investments for most users, the project's author also built a compute rental platform priced below mainstream cloud providers like Tencent Cloud and Alibaba Cloud. The platform has a distinctly anime aesthetic, using lore-flavored terms and an AI "butler" to lower the barrier to getting started.

The rental pricing gives a sense of hardware demand: RTX 5090 at roughly ¥77/day, 4090 at ¥51/day, H20 (NVLink) at ¥250/day, and AMD Strix Halo mini-PCs at ¥50/day. The creator jokes that 100 days of renting a 5090 would cost ¥7,700 — so expecting to recoup the cost of a Pro 6000 through rentals is essentially a pipe dream.

The author also developed a coding agent called Solmit that consolidates multiple agent CLI tools into a single workspace for unified scheduling — essentially a "multi-agent router." It uses powerful models like Claude and GPT as the "brain" for reasoning, while locally deployed lightweight models like Qwen3 27B handle specific task execution — smart and cost-efficient. The author's self-reported data: 85% cost savings per turn for 5-turn simple conversations, 86% for 37-turn everyday dialogues, and 92% for 246-turn large projects (not third-party verified). This follows the same logic as model-routing agents that appeared in the Japanese developer community — aggregating the strengths of different models through intelligent routing to approach frontier-model performance at a fraction of the cost.

Agent CLI (Agent Command-Line Interface) tools are a category of developer tools that emerged in recent years, with representative products including Anthropic's Claude Code and OpenAI's Codex CLI. They allow AI models to directly read and write files, execute commands, and call tools within a terminal environment, enabling autonomous completion of multi-step programming tasks. What Solmit does with "multi-agent routing" is essentially add a scheduling layer on top of these tools: based on task complexity and cost, it dynamically decides which subtasks to send to expensive cloud frontier models and which to route to locally deployed lightweight models. This architecture can dramatically reduce API costs, but also introduces latency and potential errors in the routing judgment itself.

Will Heterogeneous Acceleration Become the Norm?

GPU prices show no signs of dropping in the short term, which means various approaches to "squeezing" idle compute — orthodox or otherwise — will almost certainly proliferate. The creator himself is exploring how to use the otherwise-idle integrated GPU and NPU inside Intel CPUs to accelerate Prefill for MoE models, following the same principle: don't let any piece of hardware sit idle.

The barrier to entry for these projects remains relatively high. The most accessible combination for everyday users is still something like a DGX Spark or AMD AI Max unified-memory device. But heterogeneous acceleration points to an important direction: in an era of high hardware costs, clever architectural division of labor at the system level may offer better value than simply throwing more hardware at the problem. For users who want to run large models locally on a limited budget, keeping a close eye on these open-source projects is well worth the effort.