Ollama Local LLM Deployment Tutorial: Complete Guide from Installation to API Integration

Why Deploy Large Language Models Locally?

In our previous series, we systematically covered core concepts including calling LLM APIs with Python, streaming responses, prompt engineering, RAG knowledge bases, Function Calling, Agents, and MCP protocols. However, all previous examples relied on external LLM services (such as DeepSeek). Today we're tackling a key question — how to run open-source large language models locally using Ollama.

Local LLM deployment serves several core use cases:

• Data Privacy Protection: Hospitals, law firms, and internal corporate documents involve confidential data that cannot be sent to third-party APIs due to data breach risks
• Cost Reduction for Batch Processing: Calling external APIs for large volumes of simple tasks is extremely expensive; running locally overnight costs only electricity, with comparable results
• Reduced Inference Latency: Eliminates network round-trip delays to external LLMs; local GPU inference responds very quickly
• Eliminating External Dependencies: No longer constrained by vendor rate limiting, service outages, or price increases
• Deeper Understanding of Model Internals: Gain intuitive understanding of model parameters, quantization, and memory usage — an excellent way to learn AI

What is Ollama? The Simplest Local LLM Runtime Framework

Why Choose Ollama

Ollama is to local LLMs what Docker is to containerized deployment — incredibly easy to get started with. This analogy goes beyond stylistic similarity: Ollama borrows the core philosophy of containerized deployment, packaging the model runtime environment, dependencies, and inference engine into a unified distributable unit. Under the hood, it's built on the llama.cpp inference engine, supporting CPU acceleration via AVX2/AVX512 instruction sets, as well as NVIDIA CUDA, Apple Metal, AMD ROCm, and other GPU backends, achieving a unified interface across hardware platforms — this is the technical foundation that enables "one command to run a model." Ollama supports cross-platform operation (macOS, Linux, Windows), is easy to install and use, and is one of the most popular local LLM tools available today.

Ollama Installation

Installation methods for different operating systems:

• macOS: Simply run brew install ollama
• Windows/Linux: Download the appropriate installer from the Ollama official website

After installation, Ollama starts as a background service, listening on localhost:11434 by default.

LAN Sharing Tip: If you want to share your local model service with colleagues on the same network, change the listening address to 0.0.0.0; otherwise it will only be accessible from the local machine.

Ollama Command Quick Reference

# Check version
ollama --version

# Manually start the service
ollama serve

# Pull a model
ollama pull qwen2.5:7b

# Run a model (auto-pulls if not downloaded)
ollama run qwen2.5:1.5b

# List downloaded models
ollama list

# Delete a model
ollama rm model_name

Model files are stored by default in ~/.ollama/models under your user directory. Ollama uses the GGUF (GPT-Generated Unified Format) model format, which evolved from the llama.cpp project and is the mainstream quantization format supporting CPU and GPU hybrid inference — it has become the de facto standard in the local deployment ecosystem.

Starting the Ollama Service and Verification

If Ollama doesn't start automatically as a service, you can manually run ollama serve. Once started, verify the service is running properly:

# Method 1: curl request
curl http://localhost:11434

# Method 2: Check version (connects to running instance)
ollama --version

If the service isn't running, executing --version will show "cannot connect to a running Ollama instance." Note: if you started the service manually, don't close the terminal window.

Model Selection and Quantization Strategy Explained

Choosing the Right Model Based on Hardware

When selecting a model, you must consider your computer's specifications. Here are recommendations after 4-bit quantization:

RAM	Recommended Model Size	Example
16GB	7B and below	Qwen2.5:7B
32GB (M-series Mac)	13B	Llama3:13B
128GB+ (Ultra series)	72B	Qwen2.5:72B

Even without a dedicated GPU, Ollama can run on pure CPU with token output speeds of approximately 5-10 tokens per second — not fast, but perfectly usable.

Understanding Model Quantization: Trading Precision for Space

The essence of quantization is reducing model weights from 16-bit floating point to 4-bit or 8-bit integers, trading precision loss for dramatically reduced memory usage. This technique originates from the deep learning model compression field: modern large language models have billions of parameters, and storing a 7B parameter model at FP16 precision requires approximately 14GB of memory — extremely unfriendly to consumer hardware. Quantization maps continuous floating-point values to a limited integer range, compressing each parameter's storage space by several times, making it possible to run high-quality models on ordinary laptops. Different quantization schemes present clear trade-offs between compression ratio and precision loss.

Using the Qwen2.5:7B model as an example, here's a comparison of different quantization levels:

Quantization Level	Description	File Size	Characteristics
Q2	2-bit quantization	~2.3GB	Maximum compression, severe quality degradation
Q3	3-bit quantization	~3GB	Minimum acceptable quality
Q4	4-bit quantization	~4GB	Best balance between quality and size
Q8	8-bit quantization	~7GB	Near original precision
FP16	Original precision	~14GB	Lossless, maximum storage

To specify a quantization level, add a tag after the model name, for example:

ollama pull qwen2.5:7b-q8_0

For most developers, Q4 quantization offers the best value, maintaining good inference quality while dramatically reducing memory requirements.

Calling the Ollama Local Model API with Python

Once Ollama is deployed, you can call the local model directly from Python code via API — the approach is nearly identical to calling external LLM APIs — just point the API address to localhost:11434.

A practical tip is to use environment variables to flexibly switch between local and cloud models:

import os

# Switch between local/remote models via environment variable
if os.getenv("USE_LOCAL_MODEL"):
    base_url = "http://localhost:11434"
    model = "qwen2.5:7b"
else:
    base_url = "https://api.deepseek.com"
    model = "deepseek-chat"