#llama.cpp GGUF quantization
201 related articles
·3 分钟
Ollama + Qwen-Agent: Building a Local Private AI Assistant — From Deployment to Security Best Practices
A complete guide to building a local private AI assistant with Ollama and Qwen-Agent. Covers RAG knowledge integration, voice interaction, and permission isolation for a secure local AI Agent architecture.
阅读全文 →
·4 分钟
Heretic Uncensoring Technology Explained: Practical Applications on Jamba2 and Qwen3.5 Open-Source Models
Deep dive into Heretic uncensoring technology applied to Jamba2-Mini, Qwen3.5-9B, and 27B open-source models, exploring how refusal rates dropped from 97% to 4% and the safety debates involved.
阅读全文 →
·3 分钟
Practical Guide to Setting Up a Local AI Coding Environment on MacBook Pro M4
Complete guide to setting up a local AI coding environment on MacBook Pro M4, covering Ollama, MLX, Continue, Qwen3-Coder 30B configuration, and performance optimization strategies for 32GB RAM.
阅读全文 →
·2 分钟
DeepSeek V4 Flash Weights Open-Sourced: A New Option for Lightweight LLM Deployment
DeepSeek V4 Flash model weights reportedly open-sourced. This article analyzes its lightweight positioning, open-weight value, comparisons with closed-source models, and deployment guidance.
阅读全文 →
·2 分钟
Running Kimi K3 with 29GB of RAM: The Trade-offs and Costs of Extreme Quantization
Running Kimi K3 with 29GB RAM at just 0.5 tok/s. An in-depth analysis of extreme quantization techniques, performance trade-offs, and the impossible triangle of local LLM deployment.
阅读全文 →
·2 分钟
Running Kimi K3 in 29GB of RAM: The Trade-offs and Costs of Extreme Quantization
Running Kimi K3 with 29GB RAM at just 0.5 tok/s. A deep analysis of extreme quantization techniques, performance trade-offs, and the impossible triangle of local LLM deployment.
阅读全文 →
·3 分钟
No Python Required: A Unified Multimodal Local AI Generation CLI Tool
A CLI tool that enables local text, image, video, music, and 3D generation without Python. Explore its technical approach, advantages, limitations, and the growing trend of local AI tooling.
阅读全文 →
·3 分钟
No Python Required: A Unified Multimodal Local AI Generation CLI Tool
A CLI tool requiring no Python that supports local text, image, video, music, and 3D generation. Explore its technical approach, advantages, limitations, and the growing trend of local AI toolification.
阅读全文 →
·3 分钟
LLM Inference Speed Formula: Memory Bandwidth Determines Your tokens/sec
Why memory bandwidth (GB/s), not VRAM size, determines local LLM inference speed. Includes tokens/sec formula, GPU bandwidth comparison, and a practical card selection framework.
阅读全文 →
·3 分钟
LLM Inference Speed Formula: Memory Bandwidth Determines Your tokens/sec
Learn why memory bandwidth (GB/s)—not VRAM size—determines local LLM inference speed. Get the tokens/sec formula, GPU bandwidth comparisons, and a practical card selection hierarchy.
阅读全文 →
·2 分钟
Speech To Markdown: Local AI Voice-to-Structured Markdown Note-Taking Tool
Speech To Markdown is a free macOS/iOS app that converts voice to structured Markdown notes using local LLMs. Fully offline, no API keys needed, with global hotkey dictation.
阅读全文 →
·3 分钟
OpenWorker: A Free, Open-Source Local AI Agent That Keeps Your Data at Home
OpenWorker is a free, open-source desktop AI agent that runs locally, supports bring-your-own-model, and keeps your data private. Learn about its local-first design philosophy and key differentiators.
阅读全文 →
·3 分钟
OpenWorker: A Free, Open-Source Local AI Agent That Keeps Your Data at Home
OpenWorker is a free, open-source desktop AI agent that runs locally, supports bring-your-own-model, and keeps your data private. Learn about its design philosophy and key differentiators.
阅读全文 →
·3 分钟
Complete Guide to Running Claude Code with Ollama: API Error Troubleshooting & Local Model Optimization
Guide to running Claude Code via Ollama locally: troubleshooting API errors, output token limits, model freezes, with model selection, parameter tuning, and alternative tool recommendations.
阅读全文 →
·3 分钟
Local LLM Agent Lagging? Lowering num_gpu Is the Key
Local LLM crashing in Agent frameworks? The issue may be num_gpu set too high. Learn what num_gpu really controls (GPU layer offloading, not GPU count) and how to tune it for stable Agent performance.
阅读全文 →
·2 分钟
HuggingFace Open-Source Voice Assistant: A Complete Guide to Local Deployment
A deep dive into HuggingFace's speech-to-speech open-source project, covering its modular VAD, STT, LLM, and TTS pipeline architecture and the advantages of local deployment for privacy, cost, and latency.
阅读全文 →
·4 分钟
Colibri: Running a 744-Billion-Parameter LLM on an Ordinary Laptop
Colibri uses MoE hot-cold separation and 4-bit quantization to run 744B-parameter models like GLM 5.2 on consumer hardware. Learn about its three-tier memory architecture and speculative decoding.
阅读全文 →
·3 分钟
Apple Silicon LLM Inference Speed Benchmarks: A Complete Guide to Local Large Model Performance on Mac
In-depth analysis of Apple Silicon local LLM inference speed benchmarks covering M-series memory bandwidth, model quantization, MLX framework optimization, and Mac configuration guidance.
阅读全文 →
·3 分钟
Kimi K3 Launches on Ollama Cloud: Free Tier, User Experience, and Developer Guide
Kimi K3 officially launches on Ollama Cloud as an "extra high usage" model. This guide covers free tier quotas, cloud inference experience, technical advantages, and how developers can seamlessly call this high-performance LLM.
阅读全文 →
·3 分钟
Ollama's Extra Charges for Kimi 3 Spark Community Controversy: Should Open-Source Models Be Free to Use?
Ollama lists Kimi 3 with extra pay-per-use fees, breaking subscription expectations and sparking debate over open-weight models and AI service pricing tiers.
阅读全文 →