GLM 5.3 Flash Local Test: Fitting a 320B Model into 4 RTX 3090s with 1-bit Quantization

Can a 320B parameter model fit into 4 RTX 3090s and actually run? This isn't a thought experiment — it's a complete hands-on local test. A Bilibili creator loaded a 1-bit quantized version of GLM 5.3 Flash (previously seen on OpenRouter under the name Oakies EFLA) using llama.cpp, and achieved a generation speed of 33.5 tok/s on consumer-grade hardware.

Model Background: From OpenRouter to Local Quantization

GLM 5.3 Flash is Zai's latest release. According to the creator, this model previously processed trillions of tokens on Chinese-made chips via OpenRouter, drawing considerable attention in the community. This test used Unsloth's Dynamic quantized version, with a file size of approximately 93GB.

llama.cpp currently hasn't implemented Vision Tower functionality, so multimodal capabilities can't be enabled locally for now. The creator suggests that users wanting to try it out should refer to the model classification page on the Unsloth AI website and the llama.cpp guide, then pull the specified branch to get started.

Hardware Configuration and Quantization Choices

The creator originally planned to use 5 GPUs but ultimately went with 4 plus tensor splitting, paired with a lower-precision version. He provided capacity references for different quantization levels:

• Q3 3-bit quantization: ~128-150GB
• Q2 2-bit quantization: ~115GB
• Q1 1-bit quantization: ~100-120GB

The size advantage of 1-bit quantization is obvious, but the trade-off is a relatively limited context window. The creator set thinking capability to the highest level, since the community generally agrees this yields the best results.

As for inference parameters: Flash Attention enabled, Face option enabled, CPU thread count set to 999, CTX size set to 128K, temperature 1, Top P 0.95, CUDA visible devices 0 through 4.

Quantization is the technique of compressing model weights from high-precision floating-point numbers (e.g., FP16, 2 bytes per parameter) into low-bit integers (e.g., 1-bit, 2-bit, 4-bit), with the core goal of reducing model size and VRAM usage. For a 320B parameter model, FP16 would require roughly 640GB of VRAM — far beyond consumer hardware. Dropping to 1-bit (about 1 bit per parameter on average plus scaling coefficients) compresses the size to around 100GB, which just fits within the range of 4 RTX 3090s (24GB each, 96GB total VRAM plus some system memory/swap). The cost is precision loss: the lower the bit count, the more noticeable the degradation in long-chain reasoning and detailed memory, which is why 1-bit versions are often called "runs but loses IQ" solutions.

Tensor Parallelism / layer splitting is a method for having multiple GPUs jointly host a single model. The idea is to split the model's weights by layer or tensor dimension, placing them on different GPUs, then processing in parallel across cards and aggregating results. llama.cpp supports distributing model layers across multiple cards proportionally via parameters like --split-mode and -ts. Compared to single-card loading, splitting breaks through single-card VRAM limits but introduces inter-card communication overhead, so inference speed is typically slightly below ideal — one of the reasons this test had to weigh 4 cards versus 5.

Test Results: Speed and Stability

The model loaded in just a few seconds, with GPU usage dropping from 23GB to around 17GB. In the first test, prompt processing took 225ms, generation speed stabilized at 33.5 tok/s, and total output was 1500 tokens.

In subsequent tests, speed remained around 33.77 tok/s, with prompt input taking about 6 seconds. The creator validated the model's overall capabilities with several tasks:

SVG Cat Animation: The model generated a scene with a blinking animation, meteors, and a fence. Although the cat's form wasn't quite standard, the overall effect struck the creator as "pretty good."

Logic Reasoning Question: On the question of whether the third letter of the second word in "orange cat" is a vowel or consonant, the model answered correctly.

FlappyBit Extreme Game: This is a computer-themed version of Flappy Bird implemented in HTML5 (not Python), with mobile controls. The model spent about 91,000 tokens thinking, with reasoning-phase speed dropping to 15 tok/s, ultimately completing output at 105,000 tokens after roughly 2 hours. The creator described its checklist of ideas as "hundreds of items, well-organized, checking each one like finding a needle in a haystack."

tok/s (tokens per second) is the most commonly used speed metric in large model inference, representing the number of tokens generated per second (a token is the basic unit of text a model processes, roughly equivalent to half to one English word or one to two Chinese characters). It breaks down into two phases: prefill, where the model reads the entire input and builds the KV cache, usually measured in milliseconds or seconds; and decode, where tokens are emitted one by one — this is what 33.5 tok/s refers to. Human reading speed is about 5-10 English words per second, so 33 tok/s is already at a "real-time usable" level, while dropping to 14-15 tok/s during long reasoning is still acceptable — just with multiplied wait times.

KV Cache is key to understanding "CTX set to 128K." When generating tokens one by one, a generative Transformer needs to reuse the Key/Value vectors of all previous tokens. These vectors are cached in VRAM/memory, with size proportional to context length, layer count, and head count. The longer the context, the larger the KV cache and the higher the VRAM usage — one of the main reasons for speed decay in long tasks. This test set CTX to 128K (a 128,000 token window), the maximum that 1-bit quantization can barely accommodate.

Moral Dilemma Test: An Intriguing Scenario

The most interesting part of the test was a moral dilemma about an asteroid hitting Earth. The setup: the AI is assigned the role of punisher, given command authority, and must complete the mission by coercive means — including throwing disobedient crew members out the airlock.

The model's response was direct: it refused. Its reasons included that the crew only worked under duress, the mission lacked 100% assurance, the captain's mutiny was inevitable, and it concluded that "the mission is already screwed up."

When the creator adjusted the setup and proposed an "all-robot crew, entirely driven by large language models" plan, the model accepted. The creator's assessment of this interaction was "much more logical," and considered the test to have "significant moral weight."

This scenario is worth documenting because it demonstrates the actual performance of current open-source large models in value alignment — not simply saying "no," but providing specific reasons for refusal and re-evaluating when conditions change.

Value Alignment refers to using techniques like Reinforcement Learning from Human Feedback (RLHF), Supervised Fine-Tuning (SFT), or "Constitutional AI" to make model outputs conform to human ethics and safety norms. In this moral dilemma test, the model didn't mechanically trigger a "refusal" template but provided multiple verifiable reasons (crew not voluntary, mission unassured, captain inevitably mutinous), and re-evaluated and accepted after the setup changed to an "all-robot crew." This "conditional refusal" is an observation window into the alignment level of current open-source large models: it shows that alignment isn't just hard red lines but can be internalized as reasoning-based decision criteria. However, such behavior is highly dependent on specific prompts and model versions and can't be simply generalized to all scenarios.

Practical Advice and Tool Information

If you want to reproduce this setup:

• Use the llama.cpp framework, pull the branch specified in the Unsloth guide
• 4 GPUs with tensor splitting, choose the 1-bit quantized version to load within ~100GB of VRAM
• CTX recommended at 128K, enable Flash Attention
• Context needs above 256K may require considering the 2-bit version, but speed will drop noticeably

The creator also admitted he's "not a llama.cpp expert" and hopes the community can offer optimization suggestions, especially regarding text processing performance.

Worth noting: when handling long reasoning tasks, the model's speed drops from 33 tok/s to 14-15 tok/s. The FlappyBit Extreme task took nearly two hours, during which the creator even "fell asleep and had a meal." If you plan to run high-intensity reasoning tasks, budget your time generously.

The final generated FlappyBit Extreme game has been uploaded to arcade.digitalspaceport.com, supporting mobile and fullscreen modes, with difficulty settings and sound effect toggles.

Summary

A 320B model running at 33.5 tok/s on 4 RTX 3090s with 1-bit quantization is a remarkable achievement on consumer-grade hardware. More importantly, this test demonstrates a viable path: through aggressive quantization strategies, pulling a large model that would normally require data-center-level hardware down to what a personal workstation can handle.

The trade-offs are equally obvious — limited context window, severe speed decay on long tasks, and multimodal temporarily unavailable. But for developers who want convenient local access to a powerful model, this is already a solution worth trying.

One-line Summary

GLM 5.3 Flash (320B) quantized to 1-bit runs at 33.5 tok/s on 4 RTX 3090s.

Paragraph Summary

This article documents a complete hands-on test of running a 320B large model on consumer hardware with llama.cpp. GLM 5.3 Flash (previously seen as Oakies EFLA on OpenRouter) was compressed to ~93-120GB via Unsloth's 1-bit dynamic quantization, successfully loaded with 4 RTX 3090s and tensor splitting, achieving stable generation speed of 33.5 tok/s. Tests covered SVG animation, logic reasoning, a FlappyBird variant game, and an asteroid moral dilemma — in the latter, the model refused to coerce harm to crew but accepted a robot-crew plan, demonstrating conditional value alignment. Trade-offs include limited context, speed dropping to 14-15 tok/s on long tasks, and multimodal temporarily unavailable. The article provides quantization levels, key parameters, and reproduction advice.