#mixed-precision quantization
63 related articles
·2 min
Understanding GGUF Quantization: The Real Difference Between Q4_K_M and Q4_K_S
Detailed explanation of the core differences between GGUF model Q4_K_M and Q4_K_S: why same-Q4 files differ in size, k-quant protection strategies, quantization selection guide, and VRAM planning tips.
Read more →
·3 min
Running Local LLMs on 24GB VRAM? Your Actual Usable Memory Is Less Than You Think
A detailed breakdown of actual usable VRAM when running local LLMs on 24GB GPUs. Covers the three memory buckets — model weights, KV cache, and runtime headroom — with structured planning methods.
Read more →
·3 min
DeepSeek V4 Flash Real-World Test: The Secret Behind Spending Only $3 on 120M Tokens
A developer tested DeepSeek V4 Flash 0731, spending only $3 on 120M tokens. Learn how cache hit mechanisms slash API costs and tips for long-context optimization.
Read more →
·4 min
Quantization Illustrated: A Deep Dive into Core Compression Techniques for Large Language Models
A deep dive into LLM quantization techniques covering symmetric/asymmetric quantization, PTQ, QAT, GPTQ, AWQ, and outlier solutions for efficient model deployment.
Read more →
·3 min
NVFP4 Dynamic Quantization in Practice: W4A4 Accelerated Deployment for the Full Gemma-4 Model Family
NVFP4 dynamic quantization covers all five Gemma-4 model sizes using W4A4 mixed-precision with calibrated FP8 KV Cache, dramatically reducing VRAM usage and deployment costs for efficient inference from edge to cloud.
Read more →
·3 min
Unsloth Releases Quantized DeepSeek V4 Flash: Lossless Performance at 162GB
Unsloth releases UD dynamic quantized versions of DeepSeek V4 Flash 0731, offering six variants from 162GB lossless to 83GB extreme compression using MXFP4+BF16 mixed precision.
Read more →
·3 min
Local LLM Selection Guide: Practical Applications for Cybersecurity Purple Teams
A deep dive into how cybersecurity Purple Teams and SOC analysts can select locally deployed LLMs, covering hardware constraints, censored vs. uncensored models, specific recommendations, and RAG integration.
Read more →
·3 min
Unsloth Teams Up with Thinking Machines: Dynamic 1-bit Quantization Achieves 86% Compression, Running Multimodal LLMs in 270GB
Unsloth and Thinking Machines release dynamic 1-bit GGUF quantization for Inkling, compressing the model from 1.9TB to 270GB (86% reduction) while retaining 74.2% accuracy and adding vision/audio multimodal support.
Read more →
·4 min
Q8_K_XL vs. MXFP4 Naming Debate: The Quantization Truth That FP8 ≠ Q8_0
Deep dive into LLM quantization formats Q8_K_XL vs MXFP4, explaining why FP8 ≠ Q8_0, debunking the "8-bit is lossless" myth for local deployment users.
Read more →
·4 min
Running an 80B Large Model on Mac with Only 4.3GB of Memory: Extreme Compression Techniques Explained
Deep dive into how an 80B-parameter LLM runs on Mac with only 4.3GB memory, covering ultra-low-bit quantization, sparsity, memory mapping, and implications for privacy and edge AI.
Read more →
·3 min
Locally Deploying LLMs for Trading Bots: Hardware Selection & Model Recommendations
How to deploy LLMs locally on AMD RX 7800 XT 16GB for trading bots: ROCm ecosystem, 7B-14B model picks (Qwen2.5, Llama 3.1), Ollama/LM Studio setup, and system architecture design.
Read more →
·3 min
DeepSeek V4 Flash Released: Performance Approaching Claude at Just $0.18 per Million Tokens
DeepSeek V4 Flash launches with benchmark scores approaching Claude Opus 4.8 at just $0.18 per million output tokens. Deep analysis of performance, pricing, and industry impact.
Read more →
·3 min
LLM Inference Speed Formula: Memory Bandwidth Determines Your tokens/sec
Learn why memory bandwidth (GB/s)—not VRAM size—determines local LLM inference speed. Get the tokens/sec formula, GPU bandwidth comparisons, and a practical card selection hierarchy.
Read more →
·3 min
LLM Inference Speed Formula: Memory Bandwidth Determines Your tokens/sec
Why memory bandwidth (GB/s), not VRAM size, determines local LLM inference speed. Includes tokens/sec formula, GPU bandwidth comparison, and a practical card selection framework.
Read more →
·2 min
PlayingFild: A Smart Tab Management Chrome Extension Based on Page Content
PlayingFild is a Chrome extension that uses on-device ML to intelligently classify tabs by page content, separating work from leisure with focus timers, idle tab cleanup, and productivity analytics.
Read more →
·3 min
AMD CDNA5 Architecture Deep Dive: Technical Evolution and the AI Computing Competition Landscape
Deep analysis of AMD's CDNA5 architecture covering Chiplet packaging upgrades, HBM memory evolution, and low-precision compute optimization, examining how AMD challenges NVIDIA's AI chip dominance.
Read more →
·3 min
DeepSeek-OCR Deployment & Fine-Tuning in Practice: From vLLM Inference to Unsloth Fine-Tuning
Complete guide to DeepSeek-OCR from vLLM inference deployment and Unsloth model loading to fine-tuning, covering cloud server setup, GPU selection, and code examples — all on a single 4090 GPU.
Read more →
·5 min
Ollama Performance Degrades After Connecting to OpenWebUI? A Complete Troubleshooting Guide
Complete guide to troubleshooting Ollama performance degradation with OpenWebUI, covering num_ctx truncation, sampling parameter differences, GPU resource contention, and system prompt interference.
Read more →
·3 min
V100 GPU Showdown: 4×16G PCIe vs. 2×32G SXM — Which Gives You More Bang for Your Buck?
Benchmarking 4×V100 16G PCIe vs. 2×V100 32G SXM adapter for local LLM inference. Prefill speed, decode speed, power limits, and bandwidth bottlenecks analyzed.
Read more →
·3 min
Running a 27B Model on iPhone: How 1-bit Quantization Achieves Extreme Compression
PrismML's Bonsai compresses a 27B model from 54GB to 3.9GB, running at ~11 tokens/sec on iPhone. A deep dive into QAT, knowledge distillation, and speculative decoding.
Read more →