#LLM inference low VRAM
239 related articles
·2 min
Behind the Open-Source Model Frenzy: Who Will Provide Cheap Inference Services?
Open-source LLM weights don't equal low-cost access for developers. This article analyzes the inference service gap in open-source AI and how providers like Together AI and Groq are addressing it.
Read more →
·3 min
Self-Hosting Kimi K3: Is Spending 20% More on Hardware Worth a 20% Improvement in Task Performance?
Analysis of whether spending 20% more on hardware for self-hosting Kimi K3 to gain 20% task performance improvement is worthwhile, covering inference precision, VRAM optimization, and tiered deployment.
Read more →
·3 min
Complete Guide to Running Claude Code with Ollama: API Error Troubleshooting & Local Model Optimization
Guide to running Claude Code via Ollama locally: troubleshooting API errors, output token limits, model freezes, with model selection, parameter tuning, and alternative tool recommendations.
Read more →
·3 min
Local LLM Agent Lagging? Lowering num_gpu Is the Key
Local LLM crashing in Agent frameworks? The issue may be num_gpu set too high. Learn what num_gpu really controls (GPU layer offloading, not GPU count) and how to tune it for stable Agent performance.
Read more →
·2 min
HuggingFace Open-Source Voice Assistant: A Complete Guide to Local Deployment
A deep dive into HuggingFace's speech-to-speech open-source project, covering its modular VAD, STT, LLM, and TTS pipeline architecture and the advantages of local deployment for privacy, cost, and latency.
Read more →
·3 min
Kimi K3 vs Claude Real-World Test: Near-Equal Performance at 1/8 the Price
Real-world comparison of Kimi K3 vs Claude flagship across e-commerce pages, 3D fighting games, and flight simulators. Kimi K3 delivers 90% output quality at 1/8 the price with faster speeds and local deployment support.
Read more →
·3 min
Kimi K3 Deep Dive: How a 3 Trillion Parameter Open-Weight Model Reaches the Frontier
Deep dive into Kimi K3: the largest open-weight model at 3 trillion parameters, surpassing Opus-level models in Agentic coding with 896-expert MoE architecture, 1M token context, at Sonnet pricing.
Read more →
·4 min
Colibri: Running a 744-Billion-Parameter LLM on an Ordinary Laptop
Colibri uses MoE hot-cold separation and 4-bit quantization to run 744B-parameter models like GLM 5.2 on consumer hardware. Learn about its three-tier memory architecture and speculative decoding.
Read more →
·4 min
Laguna S 2.1 Deep Dive: Hands-On Local Deployment of a 118B Parameter Coding Model
In-depth review of Poolside's Laguna S 2.1 open-source coding model: MoE architecture, RL training, DGX Spark local deployment, and real-world agentic coding tests with 8B active parameters.
Read more →
·4 min
Kimi K2/K3 Technical Speculation: Deep Dive into RL Hyperparameters and Multi-Teacher Distillation
Deep analysis of RL hyperparameter tuning challenges and 9-policy multi-teacher distillation in Kimi K2/K3 training, exploring the shift from scale to training craft.
Read more →
·3 min
A 9B Model Beats Frontier LLMs After Just $500 in RL Fine-Tuning
Just $500 in RL fine-tuning enables a 9B open-source model to outperform frontier LLMs on catalog review tasks. Analysis of when small-model RL works and its enterprise implications.
Read more →
·3 min
From RAG to Agent Tuning: A Complete Guide to Deploying LLM Agents
A deep dive into LLM Agent frameworks covering RAG, Agent core components (tools, memory, planning), and Agent Tuning workflows with cost considerations for production deployment.
Read more →
·3 min
Kimi K3 Launches on Ollama Cloud: Free Tier, User Experience, and Developer Guide
Kimi K3 officially launches on Ollama Cloud as an "extra high usage" model. This guide covers free tier quotas, cloud inference experience, technical advantages, and how developers can seamlessly call this high-performance LLM.
Read more →
·3 min
Ollama's Extra Charges for Kimi 3 Spark Community Controversy: Should Open-Source Models Be Free to Use?
Ollama lists Kimi 3 with extra pay-per-use fees, breaking subscription expectations and sparking debate over open-weight models and AI service pricing tiers.
Read more →
·3 min
Is Ollama Pro Worth It? An In-Depth Comparison of the $20 Subscription vs. API Costs
In-depth analysis of Ollama Pro's $20/month subscription value, comparing usage quotas, equivalent API costs, and ZDR privacy policy to help developers decide if it's worth it.
Read more →
·3 min
DeepSeek-OCR Deployment & Fine-Tuning in Practice: From vLLM Inference to Unsloth Fine-Tuning
Complete guide to DeepSeek-OCR from vLLM inference deployment and Unsloth model loading to fine-tuning, covering cloud server setup, GPU selection, and code examples — all on a single 4090 GPU.
Read more →
·3 min
Deep Dive into the Kimi-K3 Technical Report: A Comprehensive Analysis of Architecture, Long-Context Processing, and Reasoning Capabilities
Deep dive into Moonshot AI's Kimi-K3 technical report, analyzing its long-context processing, MoE architecture, reasoning improvements, and its position in global AI competition.
Read more →
·3 min
Deep Dive into the Kimi-K3 Technical Report: Architecture, Long Context, and Reasoning Capabilities
Deep analysis of Moonshot AI's Kimi-K3 technical report covering long context processing, MoE architecture, reasoning capabilities, and China's position in the global AI competition.
Read more →
·2 min
RX 9060 XT vs RTX 5060 Ti: Which GPU to Pick for Local AI?
RX 9060 XT vs RTX 5060 Ti — both 16GB VRAM, but which is better for local AI? We compare CUDA ecosystem, ROCm compatibility, LLM inference, and real-world usability.
Read more →
·2 min
RX 9060 XT vs RTX 5060 Ti: How to Choose a GPU for Local AI
RX 9060 XT vs RTX 5060 Ti both offer 16GB VRAM — which is better for local AI inference? A full comparison of CUDA ecosystem, ROCm compatibility, LLM performance, and real-world usability.
Read more →