Running DeepSeek Locally: A Cost Analysis of Six 256GB VRAM Routes

Title

Running DeepSeek Locally: A Cost Analysis of Six 256GB VRAM Routes

Meta Description

How much VRAM does it take to run DeepSeek^Ad v4.1 Flash locally? Bilibili creator Kai compares six 256GB routes — Mac Studio, dual DGX Spark, Strix Halo, RTX 3090 + DDR4, eight-GPU stacking, and used V100 — breaking down how quantization bits determine real-world experience and calculating the cost of upgrading from 3-bit to 4-bit on each path.

Article Content

The barrier to running large models locally has never been the VRAM number itself, but rather "how many bits of precision your VRAM can hold." This is the point Bilibili creator Kai repeatedly emphasizes in an in-depth video, and it's also the core clue for understanding current local AI hardware purchasing.

At the start of the video, the creator describes a scenario many people have experienced: opening a dozen tabs to browse spec sheets, shopping for used server memory, watching forum arguments over a few bits, and finally realizing there's only one problem they actually want to solve — why is it so hard to run a sufficiently good model locally?

The Starting Point: Why Everyone Is Fixated on 256GB

The video's observation starts with Apple's pricing strategy. The 256GB Mac Studio is priced at $9,499, while the 512GB version is $22,299. A simple subtraction reveals that two 256GB units together cost $701 less than one 512GB unit. This pricing directly ignited community discussion: since two machines cost less in total, the number 256GB naturally became the center of everyone's discussion.

Next is the model side. DeepSeek v4.1 Flash has 552 billion total parameters, plus a lookup table called Ngram containing 196 billion parameters, with a full download of about 510GB. On the surface, this size seems hopeless for a 256GB machine. But the key is that the Ngram table only reads a few rows per token and doesn't need to reside in high-speed memory. New engines like Dwarfstar keep the 189GB lookup table on the SSD and read it on demand. After stripping this out, the core model's 2-bit build only needs 152GB, leaving nearly 100GB of headroom on a 256GB machine.

The Metric That Really Determines Experience: Bits per Weight

The creator uses an audio compression analogy that fits well: 128kbps sounds fine, 64kbps starts losing high-frequency detail, and 32kbps sounds like a phone call from a basement. Each weight of the model is stored with a fixed number of bits; the lower the bit count, the coarser the grid the values are compressed onto, and the more the model's behavior deviates from training time.

The Ansloth team did actual measurements: comparing different quantized versions with the original DeepSeek model, counting the consistency rate of next-token selection. 2-bit consistency is 91.2%, roughly one error every 11 tokens; 3-bit is 95.1%, one every 20 tokens; 4-bit is 98.5%, one every 67 tokens. Corresponding memory requirements: 2-bit about 165-200GB, 3-bit about 234-262GB, 4-bit about 298GB.

This table explains the framework of the entire video — a 256GB machine is essentially competing for the 3-bit line, and the cost of 4-bit varies enormously across different routes.

Quantization is the process of mapping model weights from high precision (e.g., 16-bit floating point) to low precision (e.g., 2/3/4-bit integers). Essentially, it uses fewer bits to represent each value, thereby greatly compressing model size and memory usage. Common post-training quantization methods like GPTQ and AWQ minimize quantization error on a small calibration dataset; more aggressive approaches like Unsloth's 1-bit build use binarization or ternarization strategies. The lower the bit count, the coarser the grid to which weights are rounded, and the greater the deviation of the model's output distribution from the original, manifesting as degraded generation quality and more logical errors. The "next-token consistency rate" mentioned in the article is a practical metric for measuring this deviation: it counts the proportion of steps where the quantized model selects the same token as the original model, reflecting behavioral differences in actual use more intuitively than perplexity.

Six Realistic Routes, Each with Different Costs

Mac Studio: The Most Hassle-Free, but with a Hidden Switch

The 256GB M5 Ultra base version is $9,499, the 80-core GPU version is $10,799, memory bandwidth is 1.2TB/s, one power cable, nearly silent. There are two problems: first, shipping takes 13 to 17 weeks; second, macOS by default only gives the GPU about three-quarters of memory, roughly 192GB. This means out of the box, a new machine can only run 2-bit models, and 3-bit capability is locked behind a terminal command. Setting the GPU memory limit to 248GB allows 3-bit to fit, but this setting resets on every reboot and needs to be written into a startup script.

In terms of speed, the engine's impact far exceeds hardware differences. The TensorFold build runs a mixed 2/3-bit model on the 80-core chip at 70-105 tokens/s write; the more common MLX route on 64 cores writes at 26-29 tokens/s and reads under 500, a nearly fivefold difference in read speed, and this gap comes almost entirely from the engine and build method, not GPU core count.

In the used market, the 256GB M3 Ultra has sold on Swappa for $6,700 to $9,200, with 819GB/s bandwidth, slower than the new model but available immediately.

macOS's GPU memory allocation mechanism differs from Linux. Under Apple Silicon's unified memory architecture, CPU and GPU share the same physical memory, but the system by default reserves about 75% of available memory for the GPU as the "recommended maximum working set size" (recommendedMaxWorkingSetSize) to prevent overall system memory pressure. For 256GB models, this means the GPU can only use about 192GB by default, and exceeding this triggers swapping or outright allocation failure. The terminal command sudo sysctl iogpu.wired_limit_mb=248000 can manually raise this limit, but the setting only applies to the current boot session and reverts to default after reboot. Therefore, it needs to be written into a LaunchDaemon or login script to persist. This limitation is part of macOS's memory management strategy, not a hardware defect, but it does constitute a software barrier easily overlooked when deploying local large models.

Dual DGX Spark: CUDA Compatibility and Fastest Prefill

A single Spark is 128GB; two connected with a 200Gb cable form a 256GB solution. NVIDIA's official price is $6,950 each, nearly double the $3,999 at launch, totaling $13,900 for two, and all major channels are out of stock.

Each Spark lets the GPU use about 121GB, giving about 260GB usable across two, enough to cross the 3-bit line and leave room for context. The community has already figured out a 2.9-bit build occupying about 196GB across two. According to Mia AI Lab's measurements, prefill is about 2,100 tokens/s, largely maintained even at 128K context length; but the first wait for a 128K prompt is about 63 seconds. For writing, single-unit bandwidth is 273GB/s; users have measured 31 tokens/s without a draft model, and with DeepSeek's DraftHead, 87 tokens/s in code scenarios and 131 tokens/s for structured output.

Pitfalls to note: the engine crashes if it uses more than about 92% of memory, so the last few GB are effectively unusable; three Sparks have an awkward split shape for this model, and those serious about it go straight to four, which requires a switch or ring networking — essentially moving the data center into the study.

Dual Strix Halo: The Most Divisive Route in the Community

AMD Ryzen AI Max Plus 395 (Strix Halo) single unit is 128GB; Framework desktop version is $3,449, Minisforum MS-S1 Max on sale for $3,639 (original $4,549), GMK Tech Evo X2 about $3,650. Two units cost roughly $6,900 to $7,800, cheaper than Mac, and they're two machines that can each do their own work.

But forums show clear disagreement on this route's effectiveness. Under Windows, the GPU defaults to 96GB, and after new driver updates it can reach about 110GB, giving 192-220GB to the GPU across two, which is 2-bit level. Under Linux, kernel settings can push each to about 120GB, nominally just crossing the 3-bit line, but users who actually tried describe almost no space left after model loading, with several crashes along the way. USB4 between machines is rated at 40 or 80Gbps, but users have measured only about 9.2Gbps on a 40Gbps link. One engine developer ran DeepSeek on two machines at about 27 tokens/s write and 700-750 tokens/s prefill; another developer analyzed that the second machine only brings about 23% write improvement for short prompts, while read for entirely new long prompts is actually 13% slower.

Worth watching is AMD's newer Gorgon Halo chip, 192GB per unit, Minisforum version at $7,399, with users already running 2-bit DeepSeek entirely in memory on it. Additionally, a developer ran a 1-bit build on a single 128GB Strix Halo at 1,000 tokens/s prefill and 23-28 tokens/s write, at the cost of Unsloth's 1-bit build erring roughly every 8 to 9 tokens.

RTX 3090 + DDR4 Server Memory: The Most Underestimated Route

This is the solution the creator says "surprised him most," and the route he believes almost no one has accounted for in the 256GB discussion. DeepSeek uses only about 16 billion of its 552 billion parameters per token, and different tokens activate different slices. The new engine approach puts the "hot" parts used by every token on the GPU, and the rest in much cheaper ordinary server memory.

Developer 0x0 built this: one RTX 3090 (a 24GB card released in 2020), paired with 256GB DDR4 memory and a 200GB SSD, running DeepSeek v4.1 Flash at 26 tokens/s single-user, 41 tokens/s with four concurrent users, prefill 600-800 tokens/s, supporting 262K context, with the builder estimating total cost at about $6,000.

The real key is the memory price difference. DDR5 server memory is heavily consumed by AI infrastructure demand, with the cheapest quotes around $29.37 per GB; while 64GB DDR4 sticks for AMD EPYC server motherboards around 2019 are $399 new, equivalent to $6.23 per GB, nearly five times cheaper. 256GB DDR5 costs about $7,500, almost the price of a Mac Studio; DDR4 is only about $1,600. This also makes 4-bit the only affordable upgrade here: adding two more 64GB sticks costs $798, and if the motherboard has empty slots, it can reach the roughly 300GB needed for 4-bit. No one has publicly released this exact build yet, but the path exists and the prices are real.

A more extreme validation: a developer ran fully unquantized DeepSeek on a 96GB RTX Pro 6000 plus 512GB DDR4, using 89GB of graphics card memory and 346GB of system memory, at 65 tokens/s single-user and 143 tokens/s with eight users. The cost is everything about old servers — louder than a Mac, software branches changing weekly, and doing server work rather than consumer work.

MoE (Mixture of Experts) architecture is the core design of the DeepSeek series. Taking DeepSeek v4.1 Flash as an example, it has 552 billion total parameters, but each token activates only about 16 billion of them — the routing network dynamically selects a few "expert" sub-networks to participate in computation based on the input. This sparse activation mechanism makes the model's actual computation and memory bandwidth requirements far lower than the total parameter count suggests. However, the weights of all experts still need to be stored somewhere; the traditional approach puts them all in GPU VRAM, resulting in extremely high VRAM requirements. The 3090 + DDR4 solution in the article exploits MoE sparsity: keeping frequently activated "hot" experts in GPU VRAM and cold experts in ordinary DDR4 memory, loaded on demand via PCIe. Since each token touches only a few experts, PCIe bandwidth pressure is manageable, replacing expensive high-speed VRAM with extremely low-cost server memory.

Eight-GPU Stacking: Strongest on Paper, Most Expensive in Reality

Eight 32GB graphics cards in one machine, real VRAM, no USB, no second machine. The problem is card prices. AMD Radeon AI Pro R9700 launched at $1,299, with some seeing it at Micro Center for $1,680 to $1,700; Intel Arc Pro B70 launched at $949, now $1,280 to $1,700. Eight of either card costs $10,240 to $13,600 just for the cards, not counting motherboard, power supply, and PCIe risers. Eight RTX 5090s at recent resale prices are about $38,000; four 72GB RTX Pro 5000s to 288GB are about $40,000.

There's one bright data point: four R9700s running Qwen 3.8 Flash reached 236 tokens/s in one user's benchmark. But DeepSeek v4.1 Flash data on eight R9700s or eight B70s has yet to be published — meaning you'd spend $13,000 on cards without seeing the one number you most want to see. And with all eight slots filled, there's no room for a ninth; 4-bit isn't adding $798 of memory but replacing the entire batch of cards — four 96GB RTX Pro 6000s at $16,000 each official price, an upgrade cost of $64,000.

Used V100/MI50: Cheapest per GB, but the Software Ecosystem Is Abandoning It

An 8-card V100 server (32GB per card, NVLink interconnect) is listed on eBay for $5,899, and RepoChad found one that sold for under $3,000, equivalent to about $23 per GB — the cheapest high-speed memory in the entire comparison. But the V100 is a 2017 chip, CUDA 13 has dropped it, and vLLM 0.20 has too. Workarounds genuinely exist and people have achieved results — for example, four 16GB V100s running Qwen 3.8 with 4-bit compressed weights reached about 219 tokens/s, matching a single RTX 5090. But specifically for DeepSeek, some have run it across three V100s plus DDR4, while other V100 users describe it as "absurdly slow" — results are highly variable.

AMD's counterpart is the MI50, also 32GB with NVLink, also abandoned by official software, but users say the ROCm community preview can run 2-bit DeepSeek on four cards at 50 tokens/s write and 275 tokens/s read. The common cost of both routes is the power consumption, noise, and heat of servers; one V100 user warned that a server in a closed room will exceed 60 degrees.

Costs Masked by Memory Arithmetic: KV Cache and First Prefill

DeepSeek v4.1 Flash's KV cache is about 890 bytes per token, about 0.93GB for a million-token context, which is indeed negligible on a 256GB machine. What's really not free is processing long context for the first time — the model must read the entire prompt before writing the first character. On Mac, TensorFold took 740 seconds, about 12 minutes, for a million-token prompt; on dual Spark, a 479K-token prompt took 280 seconds, extrapolating to about 10 minutes for a million tokens; dual Strix Halo would take at least 23 minutes based on measured speeds; the 3090 + DDR4 build takes 22 to 29 minutes. By comparison, DeepSeek's official API charges 15 cents for the same million tokens during off-peak hours.

The good news is the engine caches what it has read, so subsequent messages in the same session only process new tokens. One Mac user measured about 350 tokens/s for new turns on cached context, with a few thousand new tokens taking just seconds. So the 12 minutes is a wait at the start of a session, not between every message.

Back to That Number: The Price of the Fourth Bit

The framework the creator finally gives is clear — what to remember isn't the GB number, but "the price of the fourth bit," i.e., the cost of upgrading from 3-bit DeepSeek to 4-bit on each route: the 3090 + DDR4 build is $798 for two memory sticks; dual Strix Halo is one to two mini PCs, $3,639 to $7,800; Mac is jumping to the 512GB version for $11,500 more, or buying another 256GB and linking via Thunderbolt 5 (theoretically supported by Apple, but no one has published data on dual M5 Ultra running DeepSeek v4.1 Flash, and the second unit won't arrive until February); dual Spark is another $13,900 for two more plus a switch; eight-GPU stacking and old server cards have no empty slots, so you can only replace the entire batch, with four 96GB RTX Pro 6000s at $64,000 official price. On the API side, the fourth bit is free — because that's how it ships.

Is That Box Actually Worth It

The video does an honest accounting. A $9,499 256GB Mac Studio, at DeepSeek's off-peak output price, buys about 15.8 billion output tokens. A Mac running 24/7 at 35 tokens/s produces about 1.1 billion tokens a year, taking 14 years to break even, not counting electricity; with DraftHead pushed to 80 tokens/s, about 6 years, and at API peak prices these two numbers roughly halve. Renting compute is similar: two H200s totaling 282GB on RunPod cost about $10.58 per hour, breaking even after about 900 hours of Mac use.

The only buyer the API can't cover is the one whose data must physically stay on their own machine. The video mentions a highly upvoted comment: a client's accountant whose data is not allowed to go to the cloud or call APIs. For this group, the box's price isn't an option but a prerequisite for doing work. And when a machine runs agents for a team around the clock, the economics completely reverse — the busier the box, the more cost-effective.

The final conclusion doesn't take sides but gives scenario-based recommendations: if you need true privacy, buy the box, and the only question is which one; if 3-bit is enough and you want continuous software updates, the cheapest is the DDR4 build, $6,000, 26 tokens/s single-user, with a clear 4-bit upgrade path; if you want the most hassle-free maintenance experience, Mac is the answer — after one terminal command it's quiet, usable, and will continue to receive software support; if you want CUDA compatibility and the fastest prefill at the same price, dual Spark is the route, provided you understand that $13,900 buys 3-bit, and the fourth bit costs another $13,900; if you're willing to tinker with a mini PC stack, dual Strix Halo under Linux is quite interesting, and the 192GB single-unit Gorgon Halo is worth watching.

Choose the bit first, then find the cheapest box that can reach that bit and continue to receive software support. If none of this is worth it now, the API already runs the full four-bit model at under a dollar per million tokens — sometimes the unsexy answer is the answer.

One-line Summary

The key to running large models locally isn't the VRAM GB number, but how many bits of quantization you can run; 256GB machines are competing for the 3-bit line.

Paragraph Summary

Using Bilibili creator Kai's video as a thread, the article explores the relationship between VRAM capacity and quantization precision when running large models locally. The core point: what determines experience isn't the VRAM GB number but "bits per weight" — how many bits each weight is stored with. 256GB machines are essentially competing for the 3-bit quantization line. The article compares six realistic routes: Mac Studio, dual DGX Spark, dual Strix Halo, RTX 3090 + DDR4 server memory, eight-GPU stacking, and used V100/MI50, analyzing each one's cost, speed, upgrade path, and hidden costs. Among them, the 3090 + DDR4 route, by exploiting MoE sparsity and DDR4 price advantages, achieves 26 tokens/s at about $6,000, with a 4-bit upgrade costing only $798, making it the most underestimated solution. The article concludes with the "price of the fourth bit" framework, gives scenario-based recommendations, and points out that the API is more economical in most cases, with the real value of local deployment lying in data privacy.