#quantized model deployment
332 related articles
·4 min
Ternlight: A 7MB Browser-Side Embedding Model for Server-Free Semantic Search
Ternlight is a 7MB WebAssembly-based browser-side text embedding model requiring no server or GPU. Explore its tech, use cases, and tradeoffs for private, offline semantic search.
Read more →
·6 min
GLM 5.2 Arrives: Industry Upheaval on the Eve of AI Margin Collapse
The rise of Zhipu's GLM 5.2 is accelerating the democratization of LLM capabilities. This article analyzes the commoditization of foundation models, the logic behind margin collapse, and the opportunities and challenges facing application-layer and foundation model firms.
Read more →
·6 min
AMD MI355X Running GLM5.2: Inference Costs at Half of NVIDIA Blackwell
AMD MI355X achieves 2,626 tokens/sec throughput running GLM5.2 at half the TCO of NVIDIA Blackwell. Deep analysis of the technical logic, ROCm ecosystem progress, and market implications.
Read more →
·4 min
NVFP4 Quantization in Practice: A Guide to Compressing Large Models with NVIDIA Model Optimizer
Deep dive into NVFP4 quantization: using NVIDIA Model Optimizer to compress Nemotron 3 Ultra to FP4 checkpoints, reducing memory by 75% and boosting inference throughput on Blackwell GPUs.
Read more →
·3 min
Ornith 9B Tested: Can a 16GB Mac Mini Handle Local AI Coding?
Tested Ornith 9B on a 16GB M4 Mac Mini: LM Studio setup, tower defense game vs. 35B, and honest insights into small-model accuracy limits for local AI coding.
Read more →
The Complete Guide to Local LLM Deploy…
·2 min
The Complete Guide to Local LLM Deployment: Tools, Hardware, and Model Selection
A complete guide to locally deploying open-source LLMs: covering VRAM requirements, quantization, tools like Ollama and LM Studio, and model selection tips for Llama, Qwen, and more.
Read more →
·5 min
Defending the Right to Run Local AI: A Battle for Privacy, Freedom, and Digital Sovereignty
Local AI faces a triple threat from tightening regulation, hardware lock-downs, and commercial pressure. A deep analysis of why running open-source LLMs on your own device is a digital right worth defending.
Read more →
·3 min
Unsloth Major Update: DiffusionGemma & Gemma 4 MTP Support, 2x Inference Speed Boost
Unsloth v0.1.464-beta adds DiffusionGemma, Gemma 4 MTP, and MiniMax-M3 support, delivering ~2x inference speed boost, new Hub, RAG Q&A, tensor parallelism, and full CUDA/ROCm/Windows coverage.
Read more →
·2 min
Chrome Silently Installed a 4GB Local AI Model: The Battle for Transparency in the On-Device AI Era
Google Chrome silently downloaded a 4GB Gemini Nano AI model without user consent, sparking Hacker News debate. A deep dive into on-device AI, user rights, and transparency.
Read more →
·6 min
Codex Configuration in Practice: A Complete Guide to Accessing GPT-5.5 at Low Cost from China
A detailed guide for Chinese developers on configuring the Codex CLI AI coding tool with GPT-5.5 via API proxies, covering setup steps, efficiency gains, and security risks.
Read more →
·6 min
LangChain + MCP in Practice: A Pitfall-Avoidance Guide to LLM Selection and Local Deployment
DeepSeek R1 lacks Function Calling and JSON Output by default. Qwen3's programmable thinking modes make it the top open-source agent choice. Key LLM selection pitfalls and MCP protocol updates.
Read more →
·3 min
LLM Development Learning Path: From Zero to Enterprise-Level Projects
A complete LLM development learning roadmap covering prompt engineering, RAG, AI Agents, and fine-tuning — helping beginners master LangChain, LlamaIndex, and more.
Read more →
·6 min
A 284B Model Running on a MacBook: The Real Breakthrough and Barriers of Local DeepSeek Inference
Redis creator runs 284B-parameter DeepSeek model on a MacBook Pro at 26 tokens/sec using a pure C engine, asymmetric quantization, and MoE architecture.
Read more →
·2 min
Ollama: The 175K-Star Tool for Running Local LLMs with One Command
Ollama is an open-source local LLM runner with 175K+ GitHub Stars. Built in Go, it supports Llama, Mistral, Qwen and more — deploy in 3 steps, no setup headaches.
Read more →
·2 min
Why Banks and Hospitals Must Use Local AI: A Complete Guide to Private Large Model Deployment
Why do banks and hospitals build Local AI instead of using cloud services? This guide covers the full tech stack — Ollama, RAG, vector databases — and real-world enterprise deployment use cases.
Read more →
·3 min
Needle: Fine-Tuning a 26M Ultralight Tool-Calling Model Locally — Full Walkthrough
Needle is a 26M-parameter tool-calling model. Learn how to replace Gemini with Ollama for local training data generation and fine-tune on a single GPU, achieving 96.7% F1 — ideal for edge AI deployment.
Read more →
·7 min
AI Agent Memory Layer Comparison: Mem0, Zep, Letta, and Cloudflare — Four Major Solutions Evaluated
In-depth comparison of four AI agent memory layer solutions: Mem0's extract-retrieve approach, Zep's temporal knowledge graphs, Letta's self-editing memory, and Cloudflare Durable Objects as infrastructure primitives.
Read more →
·8 min
The Enterprise AI Flywheel: Building a Model-Eval Loop as a Competitive Moat
The core of enterprise AI isn't calling general models—it's building a self-reinforcing "model-harness-sandbox-eval" flywheel. This article analyzes the four components, tacit knowledge moats, and the "token value per watt" efficiency metric.
Read more →
·3 min
Qwythos-9B Hands-On Review: Deploying an Uncensored Open-Source Model That Inherits Claude's Reasoning Abilities
Hands-on review of Qwythos-9B, distilled from 500M+ Claude reasoning traces. Supports 1.04M token context, uncensored, runs on just 4GB VRAM. Full deployment guide included.
Read more →
·4 min
The Complete Guide to Running GLM-5.2 Locally with 4 RTX 3090s for $4,000
How to build a local AI inference server with 4 used RTX 3090 SXM4 GPUs to run GLM-5.2 via Llama.cpp and Unsloth IQ quantization, with real benchmarks on speed and quality.
Read more →