#quantized model deployment

309 related articles

Jul 20, 2026·3 min

DeepSeek and Zhipu Race to Build Custom AI Chips: China's Path to AI Compute Independence

DeepSeek seeks $7B for custom AI inference chips; Zhipu AI explores ASIC. Deep dive into China's AI compute independence strategy, multimodal generation, agents, and hardware trends.

Jul 20, 2026·3 min

Running a 27B Model on iPhone: How 1-bit Quantization Achieves Extreme Compression

PrismML's Bonsai compresses a 27B model from 54GB to 3.9GB, running at ~11 tokens/sec on iPhone. A deep dive into QAT, knowledge distillation, and speculative decoding.

Jul 20, 2026·2 min

Abliteration Explained: Building a Custom Uncensored Text Encoder for Krea 2

How a developer ran 4,000 multi-seed abliteration trials to build a 100% HarmBench-compliant uncensored text encoder for Krea 2's Qwen3-VL-4B, with five quantization formats and ComfyUI deployment guide.

Handwritten C/CUDA Inference Engine: P…

Jul 20, 2026·2 min

Handwritten C/CUDA Inference Engine: Pushing Qwen 35B to the Limits on RTX 5090

A deep dive into a handwritten C/CUDA inference engine for Qwen 35B on RTX 5090 (Blackwell), covering quantization, FlashAttention kernels, and memory optimization.

Jul 19, 2026·2 min

The Truth Behind Anthropic's Account Bans: Client Detection Logic and a Wave of AI Model Launches

Unpacking the technical truth behind Anthropic's account bans: hidden timezone and proxy detection logic sparks privacy debate. Plus Claude Sonnet 5, Linux support, and new releases from OpenAI, NVIDIA, and Google DeepMind.

Jul 19, 2026·4 min

AMD Ryzen AI Halo Real-World Test: Can a Non-Programmer Use Local AI to Write Code?

A non-programmer tests AMD Ryzen AI Halo by deploying local AI models to tackle a real dev task. After testing Ollama and Qwen3, the verdict: AI amplifies developers, it doesn't replace them.

Jul 19, 2026·3 min

Zero-Cost VibeCoding: Complete Guide to Connecting Local Ollama Models to Codex

Step-by-step guide to running local open-source models (Qwen3/Gemma) with Ollama, connecting to Codex via CC Switch for zero-token AI coding. Works on a 6GB VRAM laptop.

Jul 19, 2026·3 min

The Complete Ollama Guide: The Simplest Way to Run LLMs Locally

Ollama has 9M active developers and 170K GitHub stars. This guide covers its features, architecture, business model, and use cases for running LLMs locally.

Jul 17, 2026·2 min

When Self-Hosting Communities Get Drowned in AI Slop: Where Did the Technical Passion Go?

A Reddit post sparks broad resonance: self-hosting communities are being diluted by AI-generated apps and homogenized content. Exploring "AI slop," vibe coding, and how to preserve genuine technical passion.

Jul 17, 2026·3 min

Cherry Studio + MCP: A Practical Guide to Building AI Agents with Local Deployment

Learn how to build an automated AI agent using Cherry Studio, MCP protocol, and locally deployed models — covering DeepSeek integration, web scraping, and private knowledge base setup.

Bonsai 27B: The First 1-bit LLM That R…

Jul 17, 2026·3 min

Bonsai 27B: The First 1-bit LLM That Runs on Your Phone — A New Milestone for On-Device AI

Bonsai 27B is the first 27B-parameter LLM that runs on smartphones via 1-bit quantization, compressing to 3–4GB. We break down the tech, privacy benefits, and community debate.

Jul 17, 2026·3 min

Complete Guide to Deploying LLMs Locally on RobotCore Mini: From Ollama Installation to Web Interface Setup

A complete guide to deploying LLMs locally on RobotCore Mini using Ollama — covering model pulling, CLI verification, Web backend setup, and LAN access.

Jul 17, 2026·3 min

Cherry Studio + MCP + Ollama: The Complete Guide to Building a Local AI Agent and Private Knowledge Base

A hands-on guide to building a local AI agent and private knowledge base using Cherry Studio, MCP, and Ollama — with web scraping, report generation, and terminal control.

Jul 17, 2026·2 min

Running Qwen 122B on 8GB VRAM: A Complete Guide to Quantized Local Deployment

A Bilibili creator ran Qwen 122B with 256K context on just 8GB VRAM + 64GB RAM using llama.cpp. Full breakdown of quantization, deployment params, performance, and cost-effective alternatives.

Inkling Open-Weights Model: A New Expl…

Jul 16, 2026·2 min

Inkling Open-Weights Model: A New Exploration in the Open-Source AI Ecosystem

Inkling open-weights model released. Explore the key difference between open-weights and fully open-source AI, Inkling's positioning, and how to choose the right open-source LLM for production.

Jul 16, 2026·3 min

Bonsai-27B Extreme Quantization: Running 1-bit LLMs on Consumer Hardware

Bonsai-27B supports binary/ternary extreme quantization for 27B LLMs running on 8GB VRAM. Covers llama.cpp upstream progress, RTX 4060 benchmarks (30 t/s), and real-world limitations.

Apple M7 Ultra Leaked: Can 1.5TB Unifi…

Jul 16, 2026·2 min

Apple M7 Ultra Leaked: Can 1.5TB Unified Memory Challenge NVIDIA Blackwell?

Apple's M7 Ultra chip rumored to feature 1.5TB unified memory targeting NVIDIA Blackwell-level AI performance. A deep dive into UMA advantages, what "matching Blackwell" really means, and Apple's AI strategy.

Jul 16, 2026·3 min

Can Small Local Models Become Software Domain Experts? A RAG Architecture Deep Dive

Can small local models (1.5B–3B) become software domain experts? This article breaks down CPT, SFT, RAG, and Agent architectures, with a layered RAG-centric design for CPU-only local deployment.

Jul 16, 2026·3 min

Qwen3.5 Uncensored Version Local Deployment: Run a 35B Model on Just 6GB VRAM

Run Qwen3 35B-A3B uncensored locally on just 6GB VRAM using llama.cpp and GGUF quantization. Full deployment guide, code generation tests, and Agent integration walkthrough.

Jul 16, 2026·2 min

ExLlamaV3 v1.0.0 Official Release: Faster Inference, Lossless KV Quantization, and Streamlined Dependencies

ExLlamaV3 v1.0.0 releases with lossless KV cache quantization via kernel fusion, removal of flash-attention-2/xformers, major GEMM/GEMV gains, and broader tensor parallelism support.