llama.cpp
llama.cpp is an open-source large language model inference framework initiated by Georgi Gerganov, implemented entirely in C/C++ without dependency on external deep learning libraries. The project supports running large language models on diverse hardware including CPUs, GPUs, and Apple Silicon, offers multiple model quantization schemes to reduce memory and compute requirements, and is compatible with several mainstream open-source model formats, making it broadly applicable to local LLM deployment scenarios.
Core Facts
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
The GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
Timeline (last 90 days)
本地运行137B模型通常采用混合量化策略，如对关键层保留较高精度、对次要专家使用4-bit或3-bit量化，或采用GGUF、AWQ、GPTQ等格式配合llama.cpp、vLLM等推理框架。
CVE-2024-23605 is a metadata KV count overflow vulnerability in llama.cpp n_kv handling.
CVE-2024-21825 is a metadata array length overflow vulnerability in llama.cpp GGUF array/string parsing.
GGUF is the model container format used by mainstream inference frameworks such as llama.cpp, Ollama, and LM Studio; it itself contains no pickle serialization or embedded scripts, and opening the file does not execute code.
CVE-2026-33298 is a tensor dimension overflow vulnerability in llama.cpp ggml_nbytes().
The generation stage runs on a local llama.cpp model on the developer's machine, called via the HTTP interface on port :8080, and falls back to a template when the server is down.
AntAngelMed officially provides an FP8 version, and the community has a GGUF quantized version made by mradermacher that can be used with llama.cpp.
pricehunt uses llama.cpp and stores MoE weights in system memory (--cpu-moe), using only 4GB of VRAM, and can run on an ordinary 8GB graphics card.
LocalAI itself is not a model but an inference backend scheduling layer, and underneath it can connect to engines such as llama.cpp, gpt4all, Stable Diffusion, Whisper, and Piper.
Ollama v0.40.4 upgrades llama.cpp to version b11521, committed by jmorganca, mainly aimed at support for decision models and EmbeddingGemma 2.
40 more timeline events
All Facts (20)
llama.cpp was implemented in pure C/C++ by Georgi Gerganov, and the GGUF format packages model architecture metadata, vocabulary, and quantized weights into a single file.
90%VerifiedGPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedThe GGUF format was introduced by the llama.cpp project and is optimized for CPU inference scenarios.
90%VerifiedGGUF packages model weights, tokenizer vocabulary, hyperparameter configurations, and more into a single file, supporting multiple quantization precisions such as Q2_K, Q4_K_M, and Q8_0.
90%Verifiedllama.cpp是由Georgi Gerganov开发的纯C/C++推理引擎，支持在纯CPU环境下运行大模型
90%VerifiedThe GGUF format was released by the llama.cpp community in August 2023 as a replacement for the GGML format.
90%Verifiedllama.cpp was initiated by Georgi Gerganov in March 2023.
90%VerifiedThe GGUF format was designed by Georgi Gerganov, the founder of the llama.cpp project.
90%VerifiedGGUF（GPT-Generated Unified Format）是由llama.cpp项目定义的模型文件格式
90%VerifiedMLX is specifically optimized for Apple Silicon's unified memory architecture.
85%VerifiedGGUF has become the de facto standard format for mainstream local inference tools such as llama.cpp, Ollama, and LM Studio.
85%VerifiedOllama provides a Docker-like one-click model management experience, vLLM achieves high-throughput batch inference via PagedAttention, and LM Studio provides a graphical interface for desktop users.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verifiedllama.cpp supports hybrid CPU and GPU inference (layer offloading), allowing partial model layers to be loaded into GPU VRAM while the remaining layers are kept in memory
80%VerifiedThe llama.cpp project brought the GGUF quantization format into the mainstream, supporting multiple precision levels ranging from Q2_K to Q8_0.
80%VerifiedOllama simplifies local model deployment to a single command-line instruction and provides a local API interface compatible with OpenAI.
80%Verifiedllama.cpp supports GPU acceleration backends including Metal, CUDA, Vulkan, and SYCL.
80%Verifiedllama.cpp是由开发者Georgi Gerganov于2023年发起的开源项目，使用纯C/C++实现了LLaMA系列模型的推理
80%VerifiedOpen WebUI supports multiple backends including Ollama and the OpenAI API.
80%VerifiedvLLM, TensorRT-LLM, and llama.cpp are representative open-source frameworks in the field of LLM inference acceleration.
80%