Ollama Tutorial: Complete Guide to Running DeepSeek, Qwen, and Other LLMs Locally with One Click

Project Overview: What is Ollama

Ollama is an open-source project written in Go, designed for quickly running various large language models locally. The project has earned over 170,000 Stars on GitHub with nearly 16,000 Forks, making it the most popular local LLM runtime framework available today.

Its core philosophy is straightforward: Make running large models as simple as executing a single command. Whether you want to run Kimi-K2.5, GLM-5, MiniMax, DeepSeek^Ad, gpt-oss, Qwen, or Gemma, Ollama can help you deploy them in minutes.

Mainstream Models Supported by Ollama

Comprehensive Coverage of Chinese LLMs

Ollama's model library is particularly impressive when it comes to Chinese large models:

Kimi-K2.5: The latest model released by Moonshot AI, with outstanding benchmark performance
GLM-5: Zhipu AI's latest-generation general-purpose language model
MiniMax: Large language model from MiniMax
DeepSeek: DeepSeek's open-source model series, renowned for its cost-effectiveness
Qwen: Alibaba's Tongyi Qianwen series, covering multiple parameter sizes

International Models

gpt-oss: GPT-style models from the open-source community
Gemma: Google's lightweight open-source model

This extensive model support allows users to freely switch between and compare different models within the same framework, significantly reducing the time cost of model evaluation and selection.

Model Format and Quantization Technology

One of the key technologies enabling Ollama to efficiently run such a diverse range of large models is the GGUF (GPT-Generated Unified Format) model format. GGUF was designed by Georgi Gerganov, the author of the llama.cpp project, as a model storage format optimized specifically for CPU and hybrid CPU/GPU inference. It supports multiple quantization precisions (such as Q4_0, Q4_K_M, Q5_K_M, Q8_0, etc.), compressing model weights from FP16 (16-bit floating point) to 4-8 bit integers, which can reduce model size by 2-4x while dramatically lowering memory usage. For example, a 70B parameter model requires approximately 140GB of VRAM in FP16, but only about 40GB of memory after Q4_K_M quantization, making it possible to run on consumer-grade hardware. While quantization introduces some precision loss, modern quantization algorithms (such as GPTQ, AWQ, and llama.cpp's k-quant series) have managed to keep performance degradation within acceptable bounds.

Why Ollama Is So Popular

Minimalist User Experience

Ollama's design philosophy is "Get up and running" — start quickly, infer immediately. Users don't need to wrestle with complex environment configurations, dependency management, or model format conversions. A few commands take you from installation to conversation.

Technical Advantages of Go

The project is developed in Go, which brings several practical benefits:

High Performance: Go's concurrency model is naturally suited for handling multiple concurrent requests
Cross-Platform: Compiled binaries run directly on Windows, macOS, and Linux
Simple Deployment: Single executable file with no additional runtime dependencies

Ollama's choice of Go is a technical decision worth understanding in depth. Go was released by Google in 2009, with core design goals of solving compilation speed, concurrency handling, and deployment complexity issues in large-scale software engineering. Go's goroutine mechanism allows creating hundreds of thousands of concurrent execution units with extremely low memory overhead (each goroutine occupies only about 2KB of stack space), which is crucial for a model serving framework that needs to handle multiple inference requests simultaneously. Additionally, Go compiles to statically-linked single binaries, eliminating the dependency hell common in the Python ecosystem — users don't need to install conda, pip packages, or specific versions of CUDA toolchains to run it. However, it's worth noting that Ollama's underlying inference engine actually calls llama.cpp (a high-performance inference library written in C/C++), with Go primarily handling the upper-level API service, model management, and lifecycle scheduling.

Active Community and Rich Ecosystem

Behind the 170K+ Stars and nearly 16K Forks is a highly active developer community. Numerous third-party tools and integration solutions are built around Ollama, forming a comprehensive ecosystem. Well-known projects like Open WebUI and Continue (a VS Code AI coding plugin) use Ollama as their preferred local model backend.

Open WebUI (formerly Ollama WebUI) is one of the most representative third-party projects in the Ollama ecosystem. It provides a ChatGPT-like web interface supporting multi-model switching, conversation history management, file upload analysis, RAG integration, and more. Users can get a complete local AI assistant experience with a single Docker deployment. Continue is an open-source VS Code/JetBrains IDE plugin that uses Ollama-hosted local models as a programming assistant, providing code completion, explanation, refactoring, and other features for completely offline AI-assisted programming. Furthermore, Ollama provides an HTTP interface compatible with the OpenAI API format (default port 11434), meaning that numerous existing applications developed with the OpenAI API can seamlessly switch to local models by simply modifying the base_url, greatly reducing migration costs.

Typical Use Cases for Ollama

Daily Developer Usage

Local Development and Debugging: Quickly test different model performances locally, saving API call costs
Offline Environment Deployment: Run AI models normally in environments without network access or with limited connectivity
Privacy-Sensitive Scenarios: All data is processed locally without passing through any third-party servers

Hardware Requirements for Local Inference

Running large models locally has specific hardware requirements, with the main bottleneck being memory/VRAM capacity and bandwidth. The core operation in LLM inference is matrix multiplication, whose performance is limited by memory bandwidth rather than compute power (i.e., memory-bound rather than compute-bound). Taking a 7B parameter model as an example, after Q4 quantization it requires approximately 4-5GB of memory and can achieve 30-60 tokens/s generation speed on Apple M-series chips (unified memory bandwidth of approximately 100-400GB/s); while a 70B model requires 40GB+ of memory. On the NVIDIA GPU side, an RTX 4090 (24GB VRAM) can smoothly run most 13B-34B quantized models. Ollama supports automatic detection of available GPUs (NVIDIA CUDA, AMD ROCm, Apple Metal) and performs GPU offload of model layers — placing some model layers on the GPU for acceleration while running the remaining layers on the CPU, enabling flexible hardware utilization.

Enterprise Applications

Model Evaluation and Selection: Quickly compare multiple models' performance on specific business tasks
Internal Knowledge Base Q&A: Build enterprise private knowledge assistants using RAG technology
Edge Computing Deployment: Run lightweight models on edge devices to meet low-latency requirements

Understanding RAG Technology in Depth

RAG (Retrieval-Augmented Generation) is one of the most mainstream architectural patterns in enterprise AI applications today. Its core idea is: before the large model generates an answer, first retrieve document fragments relevant to the user's question from an external knowledge base, inject these fragments as context into the prompt, and then have the model generate answers based on real data. This approach effectively addresses two core pain points of large models — knowledge cutoff date limitations and hallucination issues. A typical RAG workflow includes: document chunking, vectorization (converting text to high-dimensional vectors via Embedding models), storage in vector databases (such as Chroma, Milvus, Qdrant), retrieving similar documents, assembling prompts, and calling the LLM to generate answers. Ollama's role in this workflow is to provide localized LLM inference and Embedding computation capabilities, which can be combined with orchestration frameworks like LangChain and LlamaIndex to build complete private RAG systems.

Trends and Summary

Judging from Ollama's pace of continuously adding model support, local LLM execution is becoming critical infrastructure for AI applications. With the integration of the latest models like Kimi-K2.5 and GLM-5, Ollama has evolved from a simple runtime tool into a bridge connecting model developers and application developers.

For developers looking to experience and deploy large models locally, Ollama provides a low-barrier, high-efficiency experimentation platform. Whether for personal learning and exploration, product prototype validation, or production deployment, it is one of the most worthwhile local LLM tools to get started with today.

Key Takeaways

Ollama is an open-source project with over 170K GitHub Stars, supporting one-click local execution of multiple mainstream large language models
Latest support includes Chinese LLMs such as Kimi-K2.5, GLM-5, MiniMax, and DeepSeek, as well as international models like Gemma
Developed in Go with llama.cpp as the underlying inference engine, offering high performance, cross-platform compatibility, and simple deployment
Supports GGUF format and multiple quantization precisions, enabling consumer-grade hardware to run large parameter models
Suitable for local development and debugging, privacy-sensitive scenarios, enterprise model evaluation, RAG knowledge bases, and more
Provides an OpenAI API-compatible interface with a rich third-party ecosystem, establishing itself as the de facto standard framework for local LLM execution