#PTQ QAT
17 related articles
·2 min
Qwen Scribe: A Deep Dive into the Local Speech Transcription Tool for Apple Silicon
Qwen Scribe is a local speech transcription tool optimized for Apple Silicon, running fully offline with Qwen models. Explore its technical features, privacy benefits, and comparison with Whisper.
Read more →
·2 min
Qwen Scribe: A Deep Dive into the Local Speech Transcription Tool for Apple Silicon
Qwen Scribe is a local speech transcription tool optimized for Apple Silicon, powered by Qwen models for fully offline use. Explore its technical features, privacy benefits, and comparison with Whisper.
Read more →
·6 min
AI Application Engineer Interview Guide: Core Topics in Model Quantization and Inference Deployment
A systematic review of must-know topics for AI Application Engineer interviews: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.
Read more →
·6 min
AI Application Engineer Interview Guide: Core Topics on Model Quantization and Inference Deployment
A systematic guide to must-know AI application engineer interview topics: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.
Read more →
·3 min
Running a 27B Model on iPhone: How 1-bit Quantization Achieves Extreme Compression
PrismML's Bonsai compresses a 27B model from 54GB to 3.9GB, running at ~11 tokens/sec on iPhone. A deep dive into QAT, knowledge distillation, and speculative decoding.
Read more →
·2 min
Apple in Talks to Acquire PrismML: Can Extreme Quantization Really Run AI on an iPhone?
Apple is reportedly in talks to acquire AI startup PrismML, whose 1-bit extreme quantization could run large models on iPhone. Community tests reveal tool-calling failures and high hallucination rates.
Read more →
·2 min
Distributed AI Systems: A Practical Engineering Guide to Training, Inference, and Production Deployment
A deep dive into distributed AI systems engineering: data/model/tensor parallelism for training, KV cache, quantization, elastic scaling for inference, and cloud deployment with Kubernetes, Ray, and DeepSpeed.
Read more →
·3 min
ComfyUI-INT4-Fast: Running Flux Models Smoothly on 6GB GPUs
ComfyUI-INT4-Fast brings W4A4 quantized inference to ComfyUI. RTX 3060 (6GB VRAM) generates 1024×1024 images in 17s. Per-layer mixed-precision routing balances speed and quality for Flux models.
Read more →
·7 min
Krea2 Tested: Does the FP8 vs BF16 Quality Gap Still Exist?
A Reddit user compared FP8 and BF16 precision on the Krea2 model and found almost no difference in image quality. This article analyzes the technical reasons behind the shrinking quantization gap and its real-world impact on VRAM usage and inference speed.
Read more →
·5 min
TensorRT Multi-Device Inference: Engineering Practices for Breaking Through Single-GPU Memory Bottlenecks
NVIDIA TensorRT now supports multi-device inference via pipeline and tensor parallelism, distributing large models across multiple GPUs to break through single-card memory limits.
Read more →
·4 min
Unsloth v0.1.463-beta: Fixes Studio Crash and llama-server Permission Issue
Unsloth v0.1.463-beta fixes a Studio crash caused by access-denied errors during llama-server service discovery. Improves stability for multi-user servers and Windows environments.
Read more →
·6 min
Made by Google Event Confirmed: New Pixel Phones with AI Capabilities Coming Soon
Google confirms the Made by Google hardware event in NYC, unveiling next-gen Pixel phones with new Tensor chips and deep Gemini AI integration. On-device AI gets a major upgrade.
Read more →
·2 min
Harvard's Open-Source Textbook cs249r: A Complete Guide to Machine Learning Systems Engineering
Harvard's open-source textbook cs249r (Machine Learning Systems) has 25,600+ GitHub stars. It covers ML systems engineering, TinyML, and MLOps — free for everyone.
Read more →
·4 min
Ternlight: A 7MB Browser-Side Embedding Model for Server-Free Semantic Search
Ternlight is a 7MB WebAssembly-based browser-side text embedding model requiring no server or GPU. Explore its tech, use cases, and tradeoffs for private, offline semantic search.
Read more →
·2 min
GPT 5.6 Internal Testing Codename Revealed, Google Pays SpaceX $920M Monthly for Computing Power
OpenAI begins GPT 5.6 Kindle Alpha internal testing with stronger base reasoning. Google partners with SpaceX at $920M/month for computing power. Gemma 4 QAT enables edge deployment, Claude Cowork doubles credits.
Read more →
Tutorials·4 min
NVIDIA Model Optimizer Post-Training Quantization (PTQ) Practical Guide
A deep dive into NVIDIA Model Optimizer's PTQ workflow, covering INT8/INT4 quantization principles, calibration methods, RTX GPU optimization, and best practices for deploying quantized LLMs on consumer GPUs.
Read more →
Tutorials·5 min
The Complete Guide to Ollama: Run DeepSeek and Other LLMs Locally with a Single Command
Ollama is an open-source tool with 170K GitHub Stars that lets you run DeepSeek, Qwen, Kimi-K2.5 and other LLMs locally with one command. Learn about its model ecosystem, advantages, and use cases.
Read more →