LiteLLM in Practice: One Proxy to Rule All Your Local and Cloud LLMs

If you run AI workloads in a Homelab, you'll eventually hit the same wall: Ollama on port 11434, llama.cpp on 8080, Text Generation WebUI on 7860, plus three separate cloud APIs from OpenAI, Anthropic, and Groq — each with its own keys, endpoints, and SDK. LiteLLM's value proposition is simple: consolidate that sprawl into a single OpenAI-compatible gateway.

Endpoint Fragmentation: The Real Pain in a Homelab

As your local AI ecosystem grows, endpoint management can quickly spiral into chaos. Before adopting LiteLLM, the typical situation looks like this: every application needs conditional branching logic for different providers, leading to duplicated code; you're juggling six or more sets of credentials and endpoints; and worst of all, there's no fallback — when Ollama goes down, your entire application crashes.

On top of that, a whole range of capabilities are simply missing: no load balancing across multiple Ollama instances, no cost visibility without manual logging, no built-in rate limiting, and no straightforward path to monitoring or distributed tracing. Each issue alone is manageable, but together they trap developers in endless infrastructure maintenance instead of building things that matter.

LiteLLM's Answer: One Proxy for Everything

LiteLLM's core promise is "one proxy to rule them all." Every application talks to a single endpoint — http://homelab:4000/v1/* — which is fully compatible with the OpenAI SDK. That means existing OpenAI client code can be pointed at LiteLLM with virtually zero changes.

The model layer is fully abstracted: when your app requests gpt-4o-mini, it might be routed to a local model or a remote API — the application doesn't need to know. Key capabilities include automatic failover (try local first, fall back to paid APIs on failure), multi-instance load balancing, built-in token counting and cost estimation, per-model or global rate limiting, and retry logic with exponential backoff and jitter.

Two Categories of Providers: Local vs. Cloud

Local providers include ollama (recommended for most use cases), llama_cpp, vllm (higher throughput but more complex to configure), and tgi. Remote providers cover openai, anthropic, groq, cohere, mistralai, and others. This unified abstraction means switching or adding a provider is a config change, not a code rewrite.

A note on vLLM: vLLM is a high-throughput inference engine developed at UC Berkeley. Its core innovation — PagedAttention memory management — can push KV cache GPU memory utilization from the typical 40–60% up to over 90%, significantly increasing concurrent throughput. The tradeoff is a higher setup bar: it requires a dedicated CUDA environment, a GPU with sufficient VRAM, and more complex launch configuration than Ollama. For Homelab use, Ollama's one-command install and automatic quantization management make it the better starting point. vLLM only becomes worthwhile when you're handling significant concurrent load — multiple users sharing the instance, or running automated pipelines. LiteLLM supports both natively; switching between them only requires changing the litellm_provider field, with no changes needed at the application layer.

Routing Strategies: Waterfall Routing Is the Real Gem

LiteLLM offers several modes for deciding where to send a request, but waterfall routing is where it shines for Homelab scenarios.

Simple List: Try in Order Until One Succeeds

The most basic strategy tries models in model_list order until one succeeds. A request might first hit Ollama's qwen2.5-coder:14b, then fall back to deepseek-r1:14b, and only reach OpenAI's gpt-4o-mini as a last resort.

Load Balancing: Distribute Across Multiple Instances

When a single Ollama instance is overloaded, you can configure a pool of members pointing to different api_base values (e.g., ollama1.local, ollama2.local, ollama3.local), each with its own rpm limit. LiteLLM distributes requests across all three instances, respecting each one's per-minute request cap to prevent overload.

Waterfall Fallback Chain: From Free Local to Expensive Cloud

This is the most practical pattern — a tiered hierarchy by capability and cost:

model_list:
  # Tier 1: Fast local
  - model_name: local_fast
    litellm_provider: ollama
    model_name: qwen2.5-coder:14b
    api_base: http://localhost:11434
    tpm: 5000
  # Tier 2: More capable local model
  - model_name: local_capable
    litellm_provider: ollama
    model_name: deepseek-r1:14b
    tpm: 3000
  # Tier 3: Cost-sensitive remote fallback
  - model_name: remote_backup
    litellm_provider: openai
    model: gpt-3.5-turbo
  # Tier 4: Premium remote for quality-critical tasks
  - model_name: remote_premium
    litellm_provider: openai
    model: gpt-4o

The logic: start with the fast local model; if it's overloaded or unsuitable, switch to the more capable local model; only reach for gpt-3.5-turbo when local options are exhausted or rate-limited; and reserve gpt-4o for complex reasoning tasks. This design makes "local-first, upgrade on demand" the default behavior — saving money while maintaining availability.

tpm vs. rpm: These are two independent rate-limiting dimensions in LiteLLM. rpm (Requests Per Minute) protects the inference server's concurrency limit; tpm (Tokens Per Minute) reflects actual compute load — a single long document summarization might count as just 1 request but consume thousands of tokens. For local Ollama instances, it's worth setting both: rpm prevents request queuing and timeout buildup, while tpm prevents one large request from monopolizing the GPU and starving other users. When a pool member hits its rate limit, LiteLLM automatically routes subsequent requests to the next available model in the list — this is how waterfall fallback triggers in practice.

Deployment Options: From Simple to Advanced

There are three deployment paths. Docker container is the recommended approach — portable and isolated. Mount your config and model directories with -v, inject API keys with -e, and expose port 4000. A docker-compose.yml is also available. Direct pip install suits developers who need to debug or modify the source: pip install litellm, then launch with litellm --config config.yaml --port 4000. Kubernetes Helm Chart is for advanced users running K8s on TrueNAS SCALE — deploy via helm install and pass model lists and secrets with --set.

Configuration Deep Dive and Real-World Scenarios

The core of config.yaml is model_list. Each entry defines the model name, provider, api_base, and optional rate-limiting parameters like tpm, rpm, and max_parallel_calls. Global settings cover default_timeout, num_retries, Redis caching, CSV usage logging, and budget controls.

Three typical scenario configurations stand out: cost-sensitive developer assistant — maximize local usage, cloud only as an emergency fallback; quality-first content generation — draft locally, refine with a cloud model (gpt-4o or Claude); budget-controlled production service — set a $50/month cap, exclude local models from budget tracking (they're "free"), and use priority weights to favor cheap or free models.

Near-Zero Integration Cost for Applications

Because LiteLLM is fully OpenAI-protocol compatible, Python apps only need to point the OpenAI SDK's base_url to http://homelab.local:4000/v1 and set api_key to any non-empty string. The model field in each request maps to the alias defined in your config (e.g., ollama_qwen_coder), and routing is handled entirely by LiteLLM. JavaScript/TypeScript works the same way, and curl testing uses standard paths like /v1/chat/completions and /v1/embeddings.

Advanced Features and Operational Considerations

LiteLLM also supports Redis response caching (especially effective for repeated requests like FAQs or code explanations), CSV usage logging with custom metadata and tags, custom request/response transformers, and deep observability through LangFuse integration — automatically tracing all calls, tracking costs by project or user, and visualizing complex chains.

On best practices: keep your config in Git, never commit secrets, use per-environment configs. Monitor health checks, error rates, p50/p95/p99 latency, and token consumption. On the security side, LiteLLM binds to 0.0.0.0 by default — put it behind a reverse proxy with real authentication, and add input validation and content filtering for any public-facing applications.

About LangFuse: LangFuse is an open-source LLM observability platform that can be self-hosted (a Docker Compose setup is sufficient). Unlike general-purpose monitoring tools like Prometheus + Grafana, LangFuse understands LLM-native concepts — it records each interaction in a Trace → Span → Generation hierarchy, showing prompt content, model responses, latency distributions, token usage, and estimated cost, with multi-dimensional filtering by user ID or project tag. Integrating LangFuse with LiteLLM only requires setting LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in your config; after that, every request through the LiteLLM gateway is automatically reported, with no changes needed in application code. This makes it invaluable for answering questions like "why did this request suddenly slow down?" or "which model cost the most last month?"

Closing Thoughts: Reducing Complexity Matters More Than Saving Money

LiteLLM's real goal isn't cost reduction (though that often follows as a side effect) — it's reducing complexity and improving reliability. By abstracting away the differing APIs, quotas, and failure modes of multiple AI providers, developers can redirect their energy toward what actually matters: building useful applications and solving real problems. Your Homelab stops being a pile of disconnected AI tools and becomes a coherent, resilient platform for experimentation. For anyone frustrated by endpoint fragmentation in their self-hosted AI setup, LiteLLM is a gateway layer worth seriously evaluating.