llama.cpp
llama.cpp is an open-source large language model inference framework initiated by Georgi Gerganov, implemented entirely in C/C++ without dependency on external deep learning libraries. The project supports running large language models on diverse hardware including CPUs, GPUs, and Apple Silicon, offers multiple model quantization schemes to reduce memory and compute requirements, and is compatible with several mainstream open-source model formats, making it broadly applicable to local LLM deployment scenarios.
Core Facts
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
The GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
Timeline (last 90 days)
Gemma 4 E2B was LoRA-fine-tuned and exported as Q4_K_M GGUF (3.4 GB), run on CPU by llama.cpp's llama-server, to draft two candidate replies and filter memories.
Inside the container, llama.cpp read the host machine's resources (60GB+ of memory, 16 cores) rather than the 4GB plan quota, causing the prompt cache to potentially grow to 8GB and trigger an OOM; three LLAMA_ARG_* environment variables were used to set it back to within 4GB.
The MoE cache causes experts to run in CUDA kernels rather than on the CPU, so outputs are no longer bit-identical; greedy selection at temperature 0 may flip, and in testing under --cpu-moe only 1 of 4 responses matched the no-cache version word for word.
Concurrent requests amplify the demand for cache slots, causing more frequent LRU evictions; on the T4, four concurrent requests achieved total throughput of 37.0 tokens/s, lower than the 43.2 tokens/s of a single request, with the hit rate dropping from 95.7% to 92.3%.
The MoE cache serves only batches of up to 32 tokens; larger batches take the old offload path.
On an RTX 5090 with a 19000 MiB cache and an 89% hit rate, decoding speed reached 67.8 tokens/s.
On the T4 with --cpu-moe plus a 10000 MiB cache, the hit rate was 95.73% for small batches of up to 8 tokens and 59.91% for batches of 9 to 32 tokens.
MoE cache hit-rate statistics are not visible by default; you need to start with -lv 4 and check the logs after stopping the service.
The MoE cache covers only ffn_*_exps tensors and does not include the router, attention, or shared experts.
Using --cpu-moe to place all experts in memory with a large cache achieves a higher cache hit rate and decoding speed than keeping --fit's automatic placement.
40 more timeline events
All Facts (20)
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%Verifiedllama.cpp was initiated by Georgi Gerganov in March 2023.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedMLX is specifically optimized for Apple Silicon's unified memory architecture.
85%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedOllama provides a Docker-like one-click model management experience, vLLM achieves high-throughput batch inference via PagedAttention, and LM Studio provides a graphical interface for desktop users.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verifiedllama.cpp supports hybrid CPU and GPU inference (layer offloading), allowing partial model layers to be loaded into GPU VRAM while the remaining layers are kept in memory
80%Verifiedllama-server is a local HTTP inference service provided by llama.cpp that implements an interface specification highly compatible with the OpenAI Chat Completions API.
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedOllama simplifies local model deployment to a single command-line instruction and provides a local API interface compatible with OpenAI.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%Verifiedllama.cpp是由开发者Georgi Gerganov于2023年发起的开源项目，使用纯C/C++实现了LLaMA系列模型的推理
80%VerifiedOpen WebUI supports multiple backends including Ollama and the OpenAI API.
80%