#PTQ

194 related articles

Jul 27, 2026·3 min

Build RAG and AI Agent Projects at Zero Cost: A Practical Guide to Open-Source Tools

Build high-quality AI projects on a budget. Learn how to use Ollama, Groq, Chroma, and other free open-source tools to build RAG systems and multi-Agent workflows from scratch.

Jul 25, 2026·6 min

How the Creator of Redis Squeezed a 284B-Parameter DeepSeek into an Ordinary Laptop

Redis creator antirez built DS4 "Dwarf Star," a pure-C inference engine, using asymmetric quantization to compress DeepSeek V4 Flash from 500GB to 80.8GB for local 128GB unified-memory deployment at 26.7 tokens/sec.

Jul 25, 2026·6 min

How the Creator of Redis Squeezed a 284-Billion-Parameter DeepSeek Into an Ordinary Laptop

Redis author antirez built the pure-C inference engine DS4 "Dwarf Star," compressing DeepSeek V4 Flash from 500GB to 80.8GB via asymmetric quantization for local deployment on 128GB unified memory at 26.7 tokens/sec.

Jul 24, 2026·6 min

AI Application Engineer Interview Guide: Core Topics in Model Quantization and Inference Deployment

A systematic review of must-know topics for AI Application Engineer interviews: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.

Jul 24, 2026·6 min

AI Application Engineer Interview Guide: Core Topics on Model Quantization and Inference Deployment

A systematic guide to must-know AI application engineer interview topics: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.

Jul 21, 2026·3 min

Claude Opus 5 Launch Imminent, Alibaba Qwen Officially Integrated into Apple Intelligence

Claude Opus 5 launches next week; Alibaba Qwen integrates into Apple Intelligence for Chinese users; 27B on-device model compressed to 3.8GB; open-source models narrow gap to closed-source by 3.3%.

Jul 21, 2026·2 min

MiniMax M3 Local Deployment: 23B Active Parameters ≠ 23GB VRAM — A Complete Guide to MoE Architecture

MiniMax M3 is a 428B MoE model. Its 23B active parameters reflect per-token compute, not VRAM needs. Learn the MoE trade-offs, quantization options, and deployment paths to avoid the most common misconception.

Jul 20, 2026·3 min

Running a 27B Model on iPhone: How 1-bit Quantization Achieves Extreme Compression

PrismML's Bonsai compresses a 27B model from 54GB to 3.9GB, running at ~11 tokens/sec on iPhone. A deep dive into QAT, knowledge distillation, and speculative decoding.

Handwritten C/CUDA Inference Engine: P…

Jul 20, 2026·2 min

Handwritten C/CUDA Inference Engine: Pushing Qwen 35B to the Limits on RTX 5090

A deep dive into a handwritten C/CUDA inference engine for Qwen 35B on RTX 5090 (Blackwell), covering quantization, FlashAttention kernels, and memory optimization.

AirLLM: How a 4GB GPU Can Run a 70B Mo…

Jul 19, 2026·2 min

AirLLM: How a 4GB GPU Can Run a 70B Model — Core Principles and Use Cases

AirLLM is an open-source project that uses layer-by-layer inference to run 70B LLMs on a single 4GB GPU. Learn how it works, its tradeoffs, and ideal use cases.

Jul 19, 2026·2 min

GLM Open-Source Model: Can a Million-Token Context Window Challenge Claude Code?

GLM open-source LLM claims 1M-token context, local deployment, and coding ability rivaling Claude Code. We break down the three key selling points and evaluate the hype vs. reality.

Jul 19, 2026·3 min

Zero-Cost VibeCoding: Complete Guide to Connecting Local Ollama Models to Codex

Step-by-step guide to running local open-source models (Qwen3/Gemma) with Ollama, connecting to Codex via CC Switch for zero-token AI coding. Works on a 6GB VRAM laptop.

Block Low-Rank Compression: A Guide to…

Jul 19, 2026·2 min

Block Low-Rank Compression: A Guide to Accelerating Large Model Inference on Memory-Constrained GPUs

Learn how Block Low-Rank (BLR) decomposition compresses large model memory usage and accelerates GPU inference, including CUDA kernel optimization and combination with quantization and pruning.

The Guardian Angels Framework: How LLM…

Jul 18, 2026·3 min

The Guardian Angels Framework: How LLM Personalization Can Achieve Both Efficiency and Data Security

The Guardian Angels framework shows how LLM personalization can achieve both productivity and data security through local deployment, differential privacy, and tiered permissions.

Bonsai 27B: The First 1-bit LLM That R…

Jul 17, 2026·3 min

Bonsai 27B: The First 1-bit LLM That Runs on Your Phone — A New Milestone for On-Device AI

Bonsai 27B is the first 27B-parameter LLM that runs on smartphones via 1-bit quantization, compressing to 3–4GB. We break down the tech, privacy benefits, and community debate.

Jul 16, 2026·2 min

ComfyUI v0.28.0: Native Video Super-Resolution, 3D Support, and PixelDiT Architecture — A Complete Breakdown

ComfyUI v0.28.0 adds SeedVR2 native video super-resolution, PixelDiT architecture, 3D Gaussian Splatting export, int4 quantization, and lip-sync integration for a major multimodal AI workflow upgrade.

Jul 16, 2026·2 min

Apple in Talks to Acquire PrismML: Can Extreme Quantization Really Run AI on an iPhone?

Apple is reportedly in talks to acquire AI startup PrismML, whose 1-bit extreme quantization could run large models on iPhone. Community tests reveal tool-calling failures and high hallucination rates.

Jul 16, 2026·4 min

Can Local Models Replace Cloud AI? Five Key Questions That Determine Success or Failure

71% of ChatGPT queries can be handled by local models — but "going local" isn't a one-step move. This guide breaks down the three tiers of local models, license traps, deployment methods, and cost logic to help you find the optimal routing strategy between local and cloud AI.

Jul 15, 2026·3 min

The Complete LLM Learning Roadmap: From Beginner to Expert — Courses and Resources Recommended

Confused by scattered LLM resources and unclear learning paths? This guide maps a complete roadmap from basics to advanced, covering Karpathy, Stanford CS224N, DeepLearning.AI, Hugging Face, plus RAG, fine-tuning, and Agent deep dives.

Jul 15, 2026·3 min

Uncensored AI Models Deep Dive: How They Work, Local Deployment, and the Truth About Self-Learning

A deep dive into uncensored AI models: how censorship is removed, whether self-learning is real, and hardware requirements for local deployment. Covers Ollama, LM Studio, Llama, quantization, and more.