#KV cache bandwidth
191 related articles
Handwritten C/CUDA Inference Engine: P…
·2 min
Handwritten C/CUDA Inference Engine: Pushing Qwen 35B to the Limits on RTX 5090
A deep dive into a handwritten C/CUDA inference engine for Qwen 35B on RTX 5090 (Blackwell), covering quantization, FlashAttention kernels, and memory optimization.
Read more →
·4 min
The Local AI Inflection Point: Edge Deployment, Multi-Model Collaboration, and the Open-Source Ecosystem
AI Engineer Summit deep dive: Local AI hits a real inflection point, driven by privacy and cost. Multi-model collaboration goes mainstream, NVIDIA + ExoLabs achieve 10x gains, open-source ecosystem accelerates.
Read more →
Block Low-Rank Compression: A Guide to…
·2 min
Block Low-Rank Compression: A Guide to Accelerating Large Model Inference on Memory-Constrained GPUs
Learn how Block Low-Rank (BLR) decomposition compresses large model memory usage and accelerates GPU inference, including CUDA kernel optimization and combination with quantization and pruning.
Read more →
·3 min
Open-Source LLM Surge: K3 and V4 Drop Together as the Marginal Cost of Intelligence Approaches Zero
Kimi K3, DeepSeek V4, Liquid, and Mistral are all dropping at once. MXFP4 quantization and MoE architecture are pushing the marginal cost of intelligence toward zero. Here's what it means.
Read more →
·2 min
Running Qwen 122B on 8GB VRAM: A Complete Guide to Quantized Local Deployment
A Bilibili creator ran Qwen 122B with 256K context on just 8GB VRAM + 64GB RAM using llama.cpp. Full breakdown of quantization, deployment params, performance, and cost-effective alternatives.
Read more →
·3 min
GPUStack in Action: Complete Guide to Day 0 Deployment of DeepSeek-V4-Flash
Deploy DeepSeek-V4-Flash DSpark on 8× H20-141G using GPUStack's SGLang backend on Day 0. Full walkthrough of Web UI config, parameter tuning, and 200 tokens/sec benchmark results.
Read more →
·2 min
ExLlamaV3 v1.0.0 Official Release: Faster Inference, Lossless KV Quantization, and Streamlined Dependencies
ExLlamaV3 v1.0.0 releases with lossless KV cache quantization via kernel fusion, removal of flash-attention-2/xformers, major GEMM/GEMV gains, and broader tensor parallelism support.
Read more →
·2 min
How Large a Local LLM Can You Run on a Laptop with 8GB VRAM? A Configuration Guide
Complete guide to running local LLMs on a laptop with 8GB VRAM: real usable memory, quantization estimation, Q4 7B/8B model recommendations, Ollama setup, GPU offloading, and agent development tips.
Read more →
·2 min
GPU Sharing in Kubernetes Production: A Deep Dive into MIG, MPS, and Time-Slicing
A deep dive into MIG, MPS, and Time-Slicing GPU sharing solutions for Kubernetes production. Compare isolation, performance, and use cases to make the right choice.
Read more →
MemStitch Zero-Copy Context Bridging: …
·2 min
MemStitch Zero-Copy Context Bridging: How It Achieves 25x TTFT Speedup in vLLM
A deep dive into how MemStitch's zero-copy context bridging achieves 25x TTFT speedup in vLLM. Covers KV Cache optimization, prefill acceleration, and practical developer value.
Read more →
Hardware-Software Co-Design: A Guide t…
·2 min
Hardware-Software Co-Design: A Guide to Building Hardware-Friendly LLM Architectures
Explore AI Model Co-Design principles and how hardware-friendly LLM architecture design — covering MoE, GQA, and FP8 quantization — optimizes the accuracy, throughput, and latency trade-off.
Read more →
Apple M7 Ultra Chip Leaked: Can 1.5TB …
·2 min
Apple M7 Ultra Chip Leaked: Can 1.5TB Unified Memory Reshape Local AI Inference?
Reddit leaks suggest Apple's M7 Ultra chip could feature up to 1.5TB unified memory. We analyze the architecture, pricing debate, bandwidth limits, and ecosystem trade-offs for local LLM inference.
Read more →
Building an eGPU for Local LLM on a $1…
·2 min
Building an eGPU for Local LLM on a $1,000 Budget: A Practical Buying and Setup Guide
Building an eGPU for local LLMs on a $1,000 budget? This guide covers GPU selection, dock costs, RTX 3090 vs 3060 value, and top tools like Ollama and llama.cpp.
Read more →
Latent Reasoning: The Next-Generation …
·3 min
Latent Reasoning: The Next-Generation AI Reasoning Paradigm Beyond Chain of Thought
Is CoT really AI 'thinking'? This deep dive covers latent reasoning's rise — Coconut, HRM, BDH — and the core trade-offs between interpretability, efficiency, and governance in high-stakes AI.
Read more →
·2 min
Distributed AI Systems: A Practical Engineering Guide to Training, Inference, and Production Deployment
A deep dive into distributed AI systems engineering: data/model/tensor parallelism for training, KV cache, quantization, elastic scaling for inference, and cloud deployment with Kubernetes, Ray, and DeepSpeed.
Read more →
How Fixing Three Bugs Made Qwen3.5-122…
·3 min
How Fixing Three Bugs Made Qwen3.5-122B Run Reliably on Mac Studio
A developer fixed three critical bugs to make Qwen3.5-122B run reliably as a daily driver on Mac Studio. Explore memory management, inference stability, and quantization precision.
Read more →
·3 min
The Rise of Diffusion Language Models: DiffusionGemma Crushes Autoregressive Models in Speed Tests
Diffusion language model DiffusionGemma dramatically outpaces autoregressive Deepseek Flash in speed tests. Explore the tech behind diffusion vs. autoregressive models and their challenges.
Read more →
·6 min
GPT-5.6 vs Grok 4.5 on the Same Day: A Packed Week of Flagship AI Model Launches
OpenAI's GPT-5.6 launches with Sawa, Terra, and Luna sub-models the same day as Musk's Grok 4.5, while Anthropic, Meta, and NVIDIA make their moves. A packed week of flagship AI launches.
Read more →
·3 min
AI Compute Prices Surge Across the Board + Rise of Two Domestic Coding Tools: The Wave of Self-Reliance Arrives
The U.S. imposes its strictest-ever export controls on top AI models, while Zhipu AI and Moonshot launch self-developed coding tools the same day—amid rising GPU and cloud compute prices. A deep dive into three trends driving cost rationality and tech autonomy.
Read more →
·4 min
From Enter to First Token: Decrypting the Full LLM Inference Pipeline (Autoregression, KV Cache & Decoding Strategies)
From pressing Enter to the first character appearing, what happens inside an LLM? This article breaks down autoregressive generation, KV cache acceleration, and decoding strategies like temperature, Top-k, and Top-p.
Read more →