#FP8 KV Cache
207 related articles
·3 min
Quantprobe: An Open-Source Tool for Running 30B LLMs on 6GB GPUs
Quantprobe is an open-source memory optimization framework that enables 30B LLMs to run at 22 tokens/s on 6GB GPUs through per-layer quantization and intelligent CPU/GPU splitting.
Read more →
·4 min
Running an 80B Large Model on Mac with Only 4.3GB of Memory: Extreme Compression Techniques Explained
Deep dive into how an 80B-parameter LLM runs on Mac with only 4.3GB memory, covering ultra-low-bit quantization, sparsity, memory mapping, and implications for privacy and edge AI.
Read more →
·4 min
RosaicLabs and Atom RTL Reveal Hidden Shifts in the x86 Ecosystem
Deep analysis of RosaicLabs, Intel Atom core RTL licensing, and 32-Tile AMX expansion — exploring x86 architecture's open licensing and customization transformation in the AI era.
Read more →
·2 min
antirez Strikes Again: ds4, a Pure C Local Inference Engine for DeepSeek 4
Redis creator antirez open-sources ds4, a pure C local inference engine for DeepSeek 4 Flash and PRO with native Metal, CUDA, and ROCm support, earning nearly 20K GitHub stars.
Read more →
·3 min
Training a 1.3B Parameter LLM from Scratch: Complete Workflow and Core Challenges
A deep dive into the complete workflow of training a 1.3B parameter LLM from scratch, covering Transformer architecture design, data preparation, and distributed training optimization.
Read more →
·3 min
Running the 1.56TB Kimi K3 Model on 8GB RAM with Pure C
A developer built a pure C99 inference engine that runs the 1.56TB Kimi K3 model on 8GB RAM using MoE sparsity and NVMe on-demand loading—no GPU, 176KB binary.
Read more →
·3 min
Building a Local AI Agent Architecture: From Scattered Scripts to Your Personal Jarvis
A practical guide to consolidating scattered automation scripts into a local AI Agent hub. Covers Function Calling, Ollama+Qwen2.5 deployment, tool orchestration architecture, and a complete implementation roadmap.
Read more →
·3 min
MI355X Running Kimi K3 Benchmark: The Logic Behind Outperforming B300 in Performance per Dollar
Deep analysis of AMD MI355X running Kimi K3 with superior cost-efficiency vs NVIDIA B300, and its implications for the AI inference hardware market.
Read more →
·3 min
Persistent State Machines: How INT4 Memory Cells Reshape LLM Attention Mechanisms
Exploring how persistent state machines with INT4-quantized memory cells reshape LLM attention, breaking KV Cache memory bottlenecks for long-context inference on edge devices and high-concurrency scenarios.
Read more →
·3 min
Why Local Agents Crash: A Real-World Report from 4,265 Actual Sessions
A developer spent a month testing 4,265 Claude Code/Codex sessions, revealing why local Agents crash on consumer hardware: tool lists consume 41% of cache, q4_0 quantization traps, and eviction strategy ceilings of only 11.88%.
Read more →
·3 min
Dual RTX 3060 Running DeepSeek V4 Flash: IQ2_M Quantized Inference Benchmarked at 3.5 tok/s
Benchmarking DeepSeek V4 Flash on dual RTX 3060 GPUs with 96GB RAM at IQ2_M quantization achieving 3.5 tokens/sec. Covers hardware choices, 2-bit quantization techniques, and local LLM deployment optimization.
Read more →
·2 min
Running Local LLMs on a 24GB Mac Mini: Model Selection & Speed Optimization Guide
24GB Mac Mini too slow for local LLMs? Learn why 14B models struggle, get 3B-8B model recommendations for Home Assistant, and discover Ollama speed optimization tips.
Read more →
·3 min
How Cheap Can It Be to Serve a 2.8 Trillion Parameter Model? A Deep Dive into Inference Economics
Deep analysis of the real cost of serving a 2.8 trillion parameter model. From MoE sparse activation to batching scale effects and inference optimization, revealing why model size and serving cost are less correlated than assumed.
Read more →
·3 min
Practical Guide to Setting Up a Local AI Coding Environment on MacBook Pro M4
Complete guide to setting up a local AI coding environment on MacBook Pro M4, covering Ollama, MLX, Continue, Qwen3-Coder 30B configuration, and performance optimization strategies for 32GB RAM.
Read more →
·2 min
DeepSeek-V4-Flash Model Analysis: A New Lightweight AI Choice Prioritizing Efficiency
In-depth analysis of DeepSeek-V4-Flash model's positioning and technical path. Exploring the lightweight trend behind the Flash naming, MLA attention, MoE architecture, and its significance for open-source AI.
Read more →
·2 min
DeepSeek-V4-Flash Model Analysis: A New Lightweight AI Choice Prioritizing Efficiency
In-depth analysis of DeepSeek-V4-Flash model's product positioning and technical approach. Examining lightweight trends through the Flash naming, MLA attention mechanism, MoE architecture evolution, and implications for the open-source AI ecosystem.
Read more →
·3 min
GPT-5.6 Slashes Prices by 80%: The Industry Logic Behind the AI Price War and Opportunities for Developers
OpenAI's GPT-5.6 series sees massive price cuts—Luna drops 80% to $0.20/M input tokens. Deep analysis of the AI price war's tech drivers, competitive landscape, and impact on developer costs and model selection.
Read more →
·3 min
GPT-5.6 Price Cut of 80%: The Industry Logic Behind the AI Price War and Opportunities for Developers
OpenAI's GPT-5.6 series sees major price cuts with Luna dropping 80% to $0.20/M input tokens. Analysis of the AI price war's technical drivers, competitive landscape, and impact on developer costs.
Read more →
·3 min
Automated Cleanup of Corrupted Media Server Files: FFmpeg Detection with Sonarr Integration
Learn how to fix corrupted media files after power outages using FFmpeg and ffprobe for automated detection, integrated with Sonarr for batch cleanup and re-downloading.
Read more →
·2 min
Grok 4.5 Hands-On Review: The Best Value for 90% of Use Cases
Reddit users share hands-on experiences with Grok 4.5, analyzing its value advantage in high-speed mode, comparing it with Fable, Sol, and other competitors, and exploring the return to rational AI tool selection.
Read more →