74B Model Squeezed into 24GB VRAM: Running Locally for Free on a Single 4090

When 74 Billion Parameters Meet 24GB of VRAM

Running large models locally has long been stuck in an awkward contradiction: models smart enough to be useful typically require over 100GB of VRAM, far beyond what ordinary computers can hold; the small models that do fit aren't smart enough to handle real agentic tasks.

According to a video posted by a tech creator on Bilibili, a project called "Aijintian Open-Source Demo Deng 3.5 74B" attempts to break this deadlock — squeezing 74 billion parameters into 24GB of VRAM, which happens to be the capacity of a single RTX 4090 or 3090. More importantly, this release doesn't just publish the weights; the training data and training methodology are disclosed as well.

The creator also offers a sobering reminder in the video: there is currently no third-party independent evaluation, and the claim of being "smarter than comparable models" is the team's own assessment.

How Does a 74B Fit into 24GB? Two Techniques Stacked Together

The core idea can be summed up in one sentence: store the knowledge of a 74B model, compute with the power of a 9B model.

Sparse Mixture-of-Experts Architecture

The 74 billion parameters are not all activated for every token — only about 9 billion parameters are activated per generation. This means the compute cost aligns with a 9B model, while the knowledge capacity aligns with a 74B model. This sparse activation strategy is the classic approach of MoE (Mixture of Experts) architectures, but achieving a 74B total with a 9B activation ratio requires careful balancing in the expert routing design.

MoE (Mixture of Experts) works by splitting a large model into multiple "expert" sub-networks. Each token passes through a router (router/gating network) and is only assigned to a few of those experts for computation. This way, the total parameter count can be very large (high knowledge capacity), but the actual computation per token depends only on the number of activated experts (low compute cost).

The key difficulty lies in routing design: if routing concentrates too heavily on a few experts, the other experts get insufficient training (expert collapse); if the distribution is too uniform, the point of sparsity is wasted. Common balancing techniques include auxiliary load-balancing loss and capacity factor. A 74B total with 9B activated means only about 12% of the total is activated each time, which is a fairly aggressive level of sparsity.

2-Bit Quantization-Aware Training

The conventional approach is to compress after training, but brute-force 2-bit compression usually severely degrades reasoning ability. This project's approach is to simulate a low-precision environment during training itself, so the model adapts to 2-bit operating conditions from the early stages.

With both techniques stacked, the total weight size is kept under 24GB. For reference, comparable dense models typically require over 140GB of VRAM.

Quantization is the technique of representing weights with fewer bits to save VRAM and bandwidth. 2-bit means each weight has only 4 possible values — an extremely high compression ratio, but at the cost of severe precision loss. Especially for tasks like reasoning and long-chain thinking, brute-force post-training quantization often makes a model "dumber."

Quantization-Aware Training (QAT) inserts simulated quantization operations (fake quantization nodes) during the training phase, so that forward propagation uses low precision while backpropagation still updates with high-precision gradients, letting the model parameters actively adapt to quantization error. After training completes, direct 2-bit deployment incurs far less performance loss than post-training quantization. The trade-off is a more complex training pipeline that is more sensitive to hyperparameters. When combined with MoE, there's an additional concern: expert routing is especially sensitive to numerical precision, and low bit-width may amplify routing errors, making the combination an engineering challenge.

Hardware Reality: 24GB Is the Floor, Not the Comfort Zone

Now for a splash of cold water. 24GB is the minimum for the weight file, not a comfortable operating value. Context length, KV cache, and runtime overhead all consume additional VRAM.

Breaking it down by GPU tier: a 24GB 4090 or 3090 just barely fits the full weights, but with limited context headroom; a 12GB GPU would require a more aggressive quantization scheme or a significant context reduction.

For users without a high-end GPU, according to the video, a free online version was also officially launched the same day for testing.

The KV cache (Key-Value Cache) is another major contributor to VRAM usage during large model inference. When generating each token, the model needs to reuse the Key and Value vectors of all previous tokens. To avoid redundant computation, these vectors are cached. The cache size is roughly proportional to "number of layers × attention head dimension × sequence length × batch size." The longer the context and the longer the conversation, the larger the usage.

For example, a 74B-class model under a long context (say 32K tokens) can have its KV cache alone consume anywhere from several GB to over a dozen GB of VRAM. Add activation values, CUDA runtime, and framework overhead, and a 24GB GPU will struggle badly with long contexts. Common mitigation techniques include KV cache quantization, Grouped Query Attention (GQA), and sliding window attention — but all involve trade-offs in quality or implementation complexity.

Getting Started and Timeline

The prerequisite is a GPU with 24GB of VRAM. The simplest path is Ollama — one command to pull the model. If you'd rather avoid the command line, just search and download through a graphical interface and start chatting directly.

Five mainstream runtimes provided support on day one, and the weights went live on October 15. The creator mentioned that just hours before release, it still couldn't be found on HuggingFace, so if you want to try it early, downloading later that day is the safer bet.

Two Paths Collide

The closed-source API approach charges per token, uploads data to the vendor's servers, and limits users to parameter tuning — a black box that can't be audited and is unavailable offline. This open-source project's path is: electricity plus hardware, zero API bill, data stays entirely local, modifiable and fine-tunable, with weights, data, and methodology all public, and fully offline-capable.

What's truly being broken through is the barrier — not how much stronger the model itself has become, but that for the first time a smart model fits into an ordinary person's GPU.

According to the team's official statement, local large model downloads have already reached 5x the total for all of last year, and local models like Qwen3 and Llama have caught up to the frontier cloud models of just a few months ago.

A Measured View: Three Things to Keep in Mind

First, there's no third-party evaluation. "Faster than most dense models and smarter than models with the same VRAM footprint" is the team's own conclusion — independent benchmarks haven't been released yet.

Second, 24GB is only the weight floor. Context, cache, and runtime overhead all require additional VRAM, so the actual requirement is higher than 24GB.

Third, open source doesn't mean parity. Closed-source frontier models still lead on independent evaluations. The value of open-source solutions lies in being free and privacy-controllable — not in comprehensively surpassing them.

Who Is This For?

Privacy-sensitive professionals — in healthcare, legal, and financial fields where data cannot leave local premises, local deployment is a hard requirement.

Agent experimenters — if you want an AI assistant to help write code, monitor tasks, or call tools, 24GB of VRAM can fit today's larger smart models.

People who already have a 24GB GPU — you can run it without any additional investment.

Those wanting to learn fine-tuning — training data and methodology are fully public, greatly lowering the barrier to reproduction.

Not for those expecting it to immediately surpass top closed-source models.

One-line Summary

A 74B-parameter model is squeezed into 24GB of VRAM via sparse MoE and 2-bit quantization-aware training, fully open-sourced.

Paragraph Summary

The article introduces a project called "Aijintian Open-Source Demo Deng 3.5 74B," which attempts to resolve the local large model dilemma of "smart ones don't fit, fitting ones aren't smart." By combining a sparse mixture-of-experts architecture (74B total parameters, only ~9B activated per pass) with 2-bit quantization-aware training, it compresses the weight size to under 24GB — exactly fitting a single RTX 4090 or 3090 — while also publishing the weights, training data, and training methodology. The article also emphasizes caution: 24GB is only the weight floor, and context and KV cache still require additional VRAM; there's currently no third-party independent evaluation, and all performance comparisons are the team's own claims; the value of open source lies in being free, privacy-controllable, and fine-tunable, not in comprehensively surpassing closed-source frontier models. It's suited for privacy-sensitive professionals, agent experimenters, those who already own a 24GB GPU, and those wanting to learn fine-tuning.