RTX 4090
A flagship consumer-grade graphics card released by NVIDIA in 2022, equipped with 24 GB of GDDR6X memory. It is among the consumer GPUs with the largest single-card VRAM capacity and is widely used for local AI model inference.
Core Facts
QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
The consumer-grade GPU RTX 4090 has a VRAM ceiling of 24 GB.
Developer Primitive reported a generation speed of 142.6 tokens/s for Qwen3 Flash Next running locally.
Timeline (last 90 days)
NVIDIA 有意在 RTX 4090 上取消了 NVLink，业界普遍认为这与数据中心 A100/H100 的产品分层策略有关。
Published latency: RTX 4090 (bf16) 8 ms for a single question, 17 ms for a 384px image, and 475 states/s batched throughput; Apple M5 Pro (MPS) 30 ms / 62 ms / 78 states/s.
Ollama local deployment measured throughput: MacBook M2 16GB approximately 15 tokens/second, RTX 3060 12GB approximately 45, RTX 4090 24GB approximately 70, 8GB laptop approximately 5
The inventory of exposed devices covers high-end accelerators such as H100, H200, B200, and Blackwell Ultra B300, as well as consumer-grade GPUs including the RTX 4090 and RTX 5090.
The 35B model can achieve 45 to 60 tokens per second on a 4090-class GPU.
When using LoRA, ordinary consumer-grade graphics cards (such as the RTX 3090/4090) are sufficient to fine-tune models with billions of parameters.
The 124M scale is the training capacity upper limit that a single consumer-grade GPU (such as an RTX 3090/4090) can handle.
2B-parameter-scale models can run on a single consumer-grade GPU (e.g., RTX 3090/4090 with 24 GB VRAM) or even in some CPU inference scenarios.
High-end Nvidia GPUs (such as the H100, A100, and consumer-grade 4090) possess thousands of CUDA cores combined with large memory bandwidth, enabling them to serve multiple inference requests simultaneously as long as VRAM capacity permits.
Unoptimized baseline token decoding speeds for running Qwen 3 (27B): RTX 5090 ~75, RTX 4090 ~46, RTX 3090 ~40, M5 Max 31.4, DGX Spark 12–17, Ryzen AI Max 11.5 tokens/s.
40 more timeline events
All Facts (20)
QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
90%VerifiedThe consumer-grade GPU RTX 4090 has a VRAM ceiling of 24 GB.
80%VerifiedDeveloper Primitive reported a generation speed of 142.6 tokens/s for Qwen3 Flash Next running locally.
65%VerifiedThis workflow can generate continuous videos up to 60 seconds long on a 12 GB GPU.
65%VerifiedThe RTX 4090 has 24 GB of VRAM, and after loading the 27B Q4 model, there is typically enough space remaining to support a context of 30,000–40,000 tokens.
65%VerifiedThe RTX 4090 has more than 16,000 CUDA cores.
65%VerifiedThe RTX 4090 has 24GB of VRAM and can smoothly run most 13B–34B quantized models
65%UnverifiedRTX 4090作为消费级旗舰显卡，拥有24GB GDDR6X显存，售价约1,600美元
85%Unverified使用Ollama本地部署时，官方推荐Qwen 2.5视觉模型（32B参数），需要至少24GB显存的GPU（如RTX 4090）才能流畅运行
85%UnverifiedRTX 4090的功耗约为300W
85%UnverifiedUnsloth通常需要配合NVIDIA GPU使用，推荐RTX 3090/4090或更高
80%Unverified移动端RTX 5090采用完整的GB203核心，拥有超过10,000个CUDA核心和16GB GDDR7显存，理论算力接近上一代桌面端RTX 4090
80%Unverified单卡RTX 4090即可运行GPT-OSS 20B，单卡A100即可运行GPT-OSS 120B
80%Unverified将模型权重从FP16压缩到INT8或INT4的量化技术，可使消费级GPU如RTX 4090在本地部署时达到30-80 TPS的实用速度
75%UnverifiedWhen fine-tuning a Llama 7B model on an RTX 4090 using QLoRA, VRAM usage is approximately 5–8GB.
70%Unverified一个70亿参数的模型在RTX 4090上的推理延迟约为50-200ms
70%Unverified批量生成100张图片在RTX 4090上大约需要15-25分钟
65%UnverifiedGenerating a single 1024×1024 image using Flux 2 on an RTX 4090 takes approximately 8–15 seconds.
60%UnverifiedRTX 4090的理论FP16算力为82.6 TFLOPS，显存带宽为1008 GB/s
60%UnverifiedNVIDIA 有意在 RTX 4090 上取消了 NVLink，业界普遍认为这与数据中心 A100/H100 的产品分层策略有关。
50%