llama.cpp
llama.cpp is an open-source large language model inference framework initiated by Georgi Gerganov, implemented entirely in C/C++ without dependency on external deep learning libraries. The project supports running large language models on diverse hardware including CPUs, GPUs, and Apple Silicon, offers multiple model quantization schemes to reduce memory and compute requirements, and is compatible with several mainstream open-source model formats, making it broadly applicable to local LLM deployment scenarios.
Core Facts
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
The GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
Timeline (last 90 days)
EmbeddingGemma 2's context window was expanded from v1's 2048 tokens to 8192, MTEB Code improved from 68.76 to 78.68, and it already supports Ollama, llama.cpp, sentence-transformers, MLX, vLLM, and browser-side transformers.js; there is currently no independent retrieval benchmark test for v2, and the SciFact score of 0.7861 belongs to v1.
When model size reaches about twice the physical memory, mmap-based inference schemes collapse due to a 'page-fault storm', with the operating system blindly scheduling pages and inference stuck on disk I/O.
Q4_K_M is a GGUF quantization format widely used in the llama.cpp community, in which weights are compressed to about 4 bits using the medium tier of the k-quant mixed quantization scheme, striking a compromise between size and precision.
For very large models, the standard llama.cpp build runs out of memory as soon as the GPU is enabled, because the CPU backend copies weights for repacking and the Vulkan backend also copies CPU-side weights into buffers the system cannot reclaim. That 2.8 tok/s figure requires a specially compiled build plus extra parameters to achieve.
The author claims Janas beats llama.cpp on all 13 compared models, under the same test conditions of the same laptop and Q4_K_M quantization.
Janas runs the 48.4GB Qwen3-Next-80B on a 32GB-memory laptop at 37 tok/s; on the same machine and same model llama.cpp reaches only 2.8 tok/s.
The MXFP4_MOE quantization scheme requires a newer version of the llama.cpp build for support.
The inference engine is the core component that actually executes model computation, such as llama.cpp, Ollama, vLLM, etc., which iterate frequently and whose interfaces and default parameters often change.
Recent new capabilities of Hermes Agent: it can manage local models by itself, and can automatically install the llama.cpp inference runtime and optimize it for the hardware (such as using Metal acceleration on Apple M-series chips).
Codex CLI integrates with the openai llama.cpp ecosystem through a background service.
40 more timeline events
All Facts (20)
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%Verifiedllama.cpp was initiated by Georgi Gerganov in March 2023.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedMLX is specifically optimized for Apple Silicon's unified memory architecture.
85%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedOllama provides a Docker-like one-click model management experience, vLLM achieves high-throughput batch inference via PagedAttention, and LM Studio provides a graphical interface for desktop users.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verifiedllama.cpp supports hybrid CPU and GPU inference (layer offloading), allowing partial model layers to be loaded into GPU VRAM while the remaining layers are kept in memory
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedOllama simplifies local model deployment to a single command-line instruction and provides a local API interface compatible with OpenAI.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%Verifiedllama.cpp是由开发者Georgi Gerganov于2023年发起的开源项目，使用纯C/C++实现了LLaMA系列模型的推理
80%VerifiedOpen WebUI supports multiple backends including Ollama and the OpenAI API.
80%VerifiedvLLM, TensorRT-LLM, and llama.cpp are representative open-source frameworks in the field of LLM inference acceleration.
80%