GGUF
GGUF (GPT-Generated Unified Format) is a binary file format for storing and distributing large language models, designed and promoted by the llama.cpp project. It encapsulates model weights, metadata, and quantization parameters in a single file and supports multiple quantization precision levels to reduce model size and memory footprint. GGUF is widely used in local inference scenarios and is compatible with multiple inference frameworks and hardware platforms.
Core Facts
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
The GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
Timeline (last 90 days)
KernelAI allows importing models in GGUF format, officially labeled as an experimental feature.
Q4_K_M is a GGUF quantization format widely used in the llama.cpp community, in which weights are compressed to about 4 bits using the medium tier of the k-quant mixed quantization scheme, striking a compromise between size and precision.
The GGUF model card's own example command sets --ctx-size 131072, which is the model's full context window and will cause memory pressure on a 16 GB machine.
A script based on the gguf package can read GGUF metadata and tally memory requirements grouped by tensor, with expert tensors named blk.N.ffn_up_exps.weight.
llama.cpp uses mmap by default to load GGUF models, loading pages from the SSD on demand into the Linux page cache via major page faults.
Glacier.Inference.Distributed adopts pipeline parallelism, splitting a GGUF-format Transformer model by layer across N machines, and does not require uniform cluster hardware specifications.
MiniCPM-V 4.7's quantization formats cover GGUF, BNB, AWQ, and GPTQ, and it adds FP8 quantization support, with the GGUF version already uploaded to Hugging Face.
llama.cpp通过GGUF格式承载量化后的模型权重，常见量化等级包括Q4_K_M、Q5_K_M、Q8_0等，级别越高精度损失越小、显存占用越大。
Dawn Chorus Pro uses the Gemma 3 1B (gemma3:1b / GGUF) model to automatically generate hiking logs, running completely offline locally via Ollama or llama-server, without requiring API keys or cloud calls.
Letter suffixes in quantized version names represent different quantization strategies; the K series uses mixed precision to retain higher precision for critical layers and generally outperforms earlier uniform quantization schemes at the same size.
40 more timeline events
All Facts (20)
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedA 7B-parameter model requires only approximately 4 GB of VRAM to run after 4-bit quantization.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedIn GGUF quantization naming, Q4_K_M denotes a 4-bit quantization, Medium-precision variant using the K-Quant grouped quantization method
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedThe GGUF format uses a key-value metadata header design with built-in runtime information such as the tokenizer vocabulary, allowing a single file to fully describe a runnable model.
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedModel quantization techniques (such as 4-bit/8-bit quantization in GGUF format) enable large models that originally required tens of GB of VRAM to run on consumer-grade GPUs or even CPUs.
80%Verifiedllama.cpp是由开发者Georgi Gerganov于2023年发起的开源项目，使用纯C/C++实现了LLaMA系列模型的推理
80%VerifiedOllama is based on the llama.cpp inference engine at its core and natively supports quantized models in GGUF format.
80%Verified4-bit quantization in GGUF format can compress model weights from 32-bit floating point to 4-bit integers, reducing model size by approximately 8 times.
80%