Running LLMs Locally: The Illusion vs. the Real Value of Open-Weight Models

Open Weight vs. Open Source: What These Terms Actually Mean

Open-weight models have been advancing at a remarkable pace. Models like GLM 5.2 and the DeepSeek series are now approaching the capabilities of top closed-source frontier models. Yet amid all the enthusiasm, prominent tech blogger Theo (founder of t3.chat) went on a passionate rant: open-weight models are great, but people obsessing over "running them locally" are actually hurting the open-source movement.

It's worth noting an important distinction: "open weight" (Open Weight) and true "open source" (Open Source) are not the same thing. Open weight simply means the model's parameter files are available for download and use — but core assets like training data, full training code, and RLHF annotation data are typically not disclosed. Truly open-source AI should meet the OSI (Open Source Initiative) definition, which includes reproducible training pipelines and fully transparent datasets. Meta's LLaMA series and the GLM series are open-weight models, not open source in the strict sense — a distinction that has real implications for legal usage rights, commercialization paths, and academic reproducibility. Understanding this difference helps us assess the true progress of the "open-source movement" more objectively.

This article draws on Theo's arguments to explore why "running a frontier-level model on your own computer" is an unrealistic fantasy today — and where the genuine value of open-weight models actually lies.

"Downloadable" ≠ "Runnable"

Theo starts by clarifying a critical point: whether a model's weights are open has nothing to do with whether you can run it at home.

Take GLM 5.2, a model he praises highly — an impressively capable open-weight model that nearly matches Opus 4.x. But how large is it exactly?

• The standard version is conservatively around 400GB
• The full-precision BF16 version reaches 1.5TB
• Even quantized and pruned versions remain around 200GB

This brings up a key technique — Quantization: compressing model weights from high-precision floats (like BF16 or FP32) into low-bit integers (like INT8, INT4, or even INT2), with the primary goal of drastically reducing VRAM usage and storage size. Take a 70-billion-parameter model: at BF16 precision, each parameter takes 2 bytes, totaling about 140GB. Quantizing to 4-bit brings that down to roughly 35GB — a 75% reduction. But quantization isn't lossless: the lower the bit-width, the more noticeable the performance degradation on complex reasoning, math, and multilingual tasks. Quantization makes large models runnable, but the gap between "runnable" and "good enough" is precisely the core issue Theo is criticizing.

The bottom line: no ordinary consumer device can run these models at full capability. They are fundamentally designed for data-center-grade hardware.

Theo emphasizes that what people call "local models" are actually small, quantized models like Gemma that barely run — and there's a massive chasm between those and truly powerful open-weight models. This is what he calls the "gap between runnable and good."

Hardware Costs: A Barrier Most Users Can't Clear

Many people assume that adding more RAM solves the problem. Theo breaks this down in detail: what actually determines inference performance is VRAM (Video RAM), not system memory.

VRAM is the high-speed memory built into the GPU, dedicated to storing model weights, KV caches, and intermediate activation values. Unlike system memory, VRAM has extremely high bandwidth to the GPU's compute units — the RTX 5090, for example, exceeds 1.7 TB/s — while CPU-to-system-memory DDR5 bandwidth is only in the hundreds of GB/s range, an order of magnitude lower. When model weights exceed VRAM capacity, the inference engine must offload data to system memory or even disk (known as "offloading"), causing inference speed to plummet from tens of tokens per second to single digits, severely impacting usability. Apple's M-series chips with their Unified Memory Architecture (UMA) are an exception: GPU and CPU share the same physical memory pool, so 128GB of unified memory can be fully utilized by the GPU — which is why high-memory MacBooks are relatively competitive for local inference.

Theo gives a painfully relatable example: even if your gaming PC has 128GB of RAM paired with an RTX 5080 (16GB VRAM) or 5090 (32GB VRAM), you still can't fit these large models. Once VRAM is saturated, no amount of additional system memory helps.

So what consumer-grade options remain? He outlines three realistic paths — each with a daunting price tag:

Three "Not Quite There" Local Deployment Options

1. High-memory MacBook: The unified memory architecture makes the 128GB version viable, but Apple recently raised prices 30–50%, and maxing out the memory costs roughly $3,000 more.
2. DGX Spark: Capable, but extremely slow (~12 TPS). Theo bluntly says his own unit "just sits there collecting dust."
3. Strix Halo (e.g., Framework Desktop): The best non-Mac consumer option, but performance is limited since it doesn't use high-end NVIDIA GPUs.

On the speed metric — TPS (Tokens Per Second) is the key benchmark for LLM inference speed. The average human reading speed is about 4–6 tokens per second; models slower than this feel noticeably laggy in interactive use. The DGX Spark's ~12 TPS is barely readable, but far too slow for fast-paced development workflows.

For anyone who wants to run these models properly, costs become absurd: a single RTX 6000 Pro Blackwell with 96GB of VRAM costs around $13,000; a tiny box built from four of those cards runs $75,000 — and can barely handle one instance at a time.

Theo's sharp conclusion: these GPUs are "probably worth more flipping as resale items than actually using them."

The Three Hard Realities: Parallelism, Electricity, and Token Efficiency

Even if we set aside the capability gap and assume a local model matches frontier performance, Theo argues there are more fundamental problems that can't be ignored.

Parallelism: You Can Only Run One Instance at a Time

Real development workflows don't run just one agent. Theo says his own work involves anywhere from 1 to 40 agents running simultaneously — multiple threads, subtasks split across them, Claude and Codex used in parallel. Even if you build a system capable of running GLM 5.2 once, can you run it ten times simultaneously? Absolutely not.

There's a deep technical reason for this. The KV Cache (Key-Value Cache) is a critical mechanism in Transformer architectures for accelerating autoregressive generation: when generating each new token, the model caches the Key and Value matrices from previous computations for reuse, avoiding redundant attention calculations over all historical tokens. However, KV cache size scales with context length, number of layers, and model dimensionality — for models with hundreds of layers, the KV cache for a single long conversation can occupy several GB of VRAM. When running multiple agents concurrently, each parallel request needs its own independent KV cache space, and VRAM requirements stack linearly. vLLM's PagedAttention technology manages KV cache fragmentation more efficiently, significantly improving GPU utilization — but it still can't solve the fundamental constraint of limited total VRAM on a single consumer-grade device.

When hardware sits idle, it's pure wasted money. When demand spikes beyond capacity, you're bottlenecked. He dismisses the "reassuring talk" about vLLM caching and batching as not actually solving this problem.

Electricity Costs: Owning Hardware Doesn't Mean Free Usage

Theo had GLM 5.2 calculate this on the spot: running an RTX 5090 24/7 in San Francisco costs roughly $5 per day, or nearly $2,000 per year — and that's just one GPU. "Owning hardware doesn't mean using it for free."

Token Efficiency: Cheap Per-Token Pricing Doesn't Mean Cheaper in Practice

This is the most overlooked point. Open-weight models often "burn" tokens at a much higher rate. Using the DeepSWE benchmark as an example:

• Opus 4.8 costs roughly $8 per run
• GLM 5.2 Max costs roughly $4 per run

On the surface, GLM's per-million-token price is nearly 10x cheaper ($3 vs. $25). But because it generates far more tokens than frontier models, the actual cost difference is only about 2x. Worse, generating 3x the tokens means that even at 20% faster speed, the overall workflow is still slower.

Additionally, GLM 5.2 doesn't even have vision capabilities — it can't read screenshots or complete UI feedback loops, which are now standard features in frontier models.

The Real Value of Open Weights: Driving Competition, Not Local Deployment

Despite all the criticism, Theo also articulates what he sees as the correct use case for open-weight models. He quotes a comment he considers spot-on:

"The greatest value of open-source models isn't running them locally — it's creating competition at the provider and hardware level."

No provider can offer Opus at a low price due to licensing restrictions, but anyone can host GLM 5.2 wherever electricity is cheaper. On platforms like OpenRouter — a unified AI model API aggregation service that lets developers access hundreds of models from different providers through a single interface, with transparent pricing and automatic routing to the optimal provider based on load — the same GLM 5.2 model might be hosted by a dozen different inference providers. Some offer blazing-fast versions at 115 tokens per second; others are slower but cheaper. Users can freely trade off speed, cost, and reliability. This is precisely how open weights break the monopoly of single-vendor licensing and drive market competition in inference services.

Theo points out that nearly every pain point of running large models locally can be solved through cloud hosting:

• The runnable-vs-good gap → Cloud has sufficient hardware capacity
• High hardware costs → Converted into pay-as-you-go usage
• Limited parallelism → Cloud scales elastically
• Electricity costs → Baked into token pricing, and data centers can choose low-cost energy regions

The only thing lost is the privacy advantage of "no one can see what you're doing." For that, Theo places his hopes in future Secure Compute technologies — including Trusted Execution Environments (TEEs, such as Intel SGX and AMD SEV), homomorphic encryption, and multi-party computation. These technologies are still in early stages for AI inference, but with NVIDIA's Hopper architecture introducing confidential computing support, Theo's optimism isn't far-fetched. In the meantime, local deployment remains the only reliable option for privacy-sensitive use cases — medical, legal, enterprise internal data, and so on.

Running LLMs on Phones? Entry-Level Chips Are Going Nowhere

For those who believe "devices keep getting more powerful, so eventually everyone can run everything locally," Theo pushes back with data: while high-end iPhones and flagship Xiaomi devices have improved over the past three years, the performance of budget $200 Android chips has shown virtually zero growth — and may have slightly declined. Combined with rising memory prices and supply chain pressures, this isn't going to improve anytime soon.

He does acknowledge that lightweight on-device tasks — like Gemma 3n summarizing messages or weather updates — have genuine value on phones, both for privacy and to avoid uploading data. But running large models on a phone just drains the battery and causes overheating. "It's nowhere near as good as just calling an API."

Conclusion: Don't Let Fantasies Hurt the Open-Source Movement

Theo repeatedly emphasizes that he is a staunch advocate for open-weight models — t3.chat was born out of his love for the DeepSeek series. DeepSeek R1 once single-handedly pushed the entire industry forward, and that is the irreplaceable significance of open weights.

What he truly opposes is the fantasy-driven narrative that "you can run frontier-lab-level results on your own GPU, freeing yourself from Anthropic and OpenAI."

"If you just love building GPU clusters and playing with these models, that's awesome — go for it. But if you're pretending this is the path everyone must take, then you're deceiving people."

The core message of this passionate rant is actually quite clear: We should be able to honestly discuss how good open-weight models are, without misleading people with the lie that 'you can run them at home.'

Key Takeaways