#推理优化
61 related articles
KVFetch: Filling the Missing Temporal Recall Channel in KV Cache Compression
New arXiv paper KVFetch identifies sequential forgetting in KV cache compression and proposes a training-free temporal recall channel, restoring verbatim copy scores from 0.8 to 78.4 on RULER-16K.
AttSVD: Attention-Guided SVD Compresses KV Cache to Half the Memory Without Accuracy Loss
AttSVD compresses KV cache via attention-guided SVD along the feature axis — no training required, 50% memory, full accuracy on LongBench and agentic benchmarks.
The GPT-6.1 Sol "Leak" Mystery: Looped and Nested Model Architectures Come Into Focus
A Reddit post reverse-engineers GPT-6/6.1 Sol, Astra, and Luna architectures from inference speed data, revealing looped transformers and nested models as likely strategies for large-scale, low-cost deployment.
The Hidden Cost of Inference Latency: Are You Overlooking Model Load Time?
Tests show model loading can take 13.9s vs 1.6s for inference itself — a 10x gap. This article breaks down model availability latency, state observability, and three production deployment strategies.
GLM-5.3 Hits 469 tok/s on 8×MI355X: A Deep Dive into vLLM + TileRT Disaggregated Inference
TileRT and AMD achieve 469 tok/s single-user decode for GLM-5.3 on 8×MI355X using a disaggregated architecture where vLLM handles prefill and TileRT handles latency-critical decode.
Canary Rollouts: A Practical Guide to Zero-Downtime AI Model Upgrades in Production
Canary rollouts enable zero-downtime AI model upgrades in production. Learn how phased traffic ramping, metric gates, and automatic rollback reduce risk in dedicated inference deployments.
The Rise of Overfit Inference Engines: How Specialization Is Reshaping Local AI Deployment
Overfit inference engines like Strata, ninfer, and DwarfStar are abandoning generality to maximize performance on specific hardware. Here's what that means for local AI.
Why Real-Time AI Inference Costs 56x More Than It Should — And How Persistent Kernels Fix It
Why does serving real-time AI cost 56x more than it should? We break down continuous vs. request-based inference, and how persistent GPU kernels, synchronous batching, and compile-time scheduling fix it.
Fireworks Launches Ember-1: Shorter Reasoning, 40% Fewer Tokens
Fireworks Research releases Ember-1, a reasoning model built on Kimi K3 that cuts token usage by ~40% while maintaining top-tier quality. Here's why it matters.
TPU Megakernel Achieves 709 tokens/s on Kimi K3, Outpacing GB200
The @inferact team open-sources a TPU megakernel for Kimi K3 achieving 709 tokens/s, beating GB200's 450 tokens/s via 92-layer MoE fusion and cross-layer weight prefetching in a single Pallas kernel.
16 FPS on a Single RTX 5090: Optimizing LingBot-World 2.0 for Real-Time Inference
A developer optimizes LingBot-World 2.0 1.3B to 16 FPS on a single RTX 5090 — 2.5× faster than SGLang and 1.9× faster than NVIDIA FlashDreams — using low-precision ops, SageAttention, and custom CUDA kernels.
vLLM Deployment of Kimi K3 Achieves 2.2–2.8x Throughput Gain: A Performance Optimization Breakdown
Kimi K3 on vLLM achieves 2.2–2.8x throughput gains on B300 benchmarks over v0.27.1, driven by optimizations in scheduling, KDA state handling, and MoE kernels.
Inference Networks: Optimizing LLM Inference Costs with Threshold Policies
A new arXiv paper proposes the Inference Networks graph framework for optimizing LLM inference costs, proving that optimal model activation follows a threshold structure for significant cost reduction.
Jev: A New Frontier Model Claiming 40–400x Lower Cost and 20–200x Faster Inference
New frontier model Jev claims 40–400x lower cost and 20–200x faster inference. We break down the efficiency claims, likely technical approaches, and how to evaluate them.
Understanding KV Cache: The Space-Time Tradeoff Behind LLM Inference Optimization
A deep dive into KV Cache for LLM inference: how it eliminates redundant Key/Value recomputation in autoregressive generation, with complexity analysis and memory estimation formulas.
vLLM Full-Stack Optimization in Practice: Handling the Serving Pressure of Real-World Agent Traffic
vLLM's full-stack optimization for real-world agent traffic covers architecture, framework, and runtime layers, benchmarked on SemiAnalysis's public AgentX workload.
After Day-0 Support: How the Community Achieved 3–4x Inference Speedup for MiniMax M3
AMD and EmbeddedLLM detail how they achieved 3–4x throughput gains for MiniMax M3 on Instinct MI355X using vLLM, with a "follow the bottleneck" optimization approach.
TutorialsDeepSeek V4 Flash MTP Speculative Decoding Real-World Test: A Guide to 20% Faster Local Inference
Real-world testing of DeepSeek V4 Flash with MTP speculative decoding: ~20% speedup for code generation, minimal gains for text. Covers memory overhead, accuracy differences, Q4 vs Q3 quantization, and full deployment tutorial.
Industry InsightsNVIDIA Dynamo Snapshot: A Snapshot Recovery Solution for GPU Inference Cold Start Problems
Deep dive into how NVIDIA Dynamo Snapshot reduces LLM inference cold start time from minutes to seconds via GPU state snapshot and recovery, covering Kubernetes integration and elastic inference.
Tech FrontiersGemini 3.5 Flash Released: How Google Balances AI Model Speed and Capability
Google releases Gemini 3.5 Flash, optimizing the balance between speed and capability. Analysis of Flash series evolution, comparisons with GPT-4o mini, and practical value for developers.