#推理优化
61 related articles
Understanding vLLM and SGLang from First Principles: Deconstructing the Core Technologies of Inference Engines
Deconstructing vLLM and SGLang inference engines from first principles: how PagedAttention reduces memory fragmentation, how continuous batching boosts throughput, how RadixAttention and DSL scheduling optimize prefix reuse and complex generation workflows, plus selection advice.
TensorRT Global Performance Tuner: Breaking Past Single-Build-Route Performance Optimization
An analysis of NVIDIA TensorRT Global Performance Tuner: why single build routes can't reach hardware limits, how global tuning turns compilation from manual trial and error into configurable search, and its impact on inference deployment pipelines.
Kimi K3.1 Pro Dual Versions Debut: Highspeed Edition Steals the Spotlight
Kimi K3.1 Pro and K3.1 Pro Highspeed spark Hacker News debate. We break down the standard vs. highspeed trade-offs, community concerns over transparency and pricing, and practical guidance on model selection and architecture routing.
DeepSeek Open-Sources Inference Engine DeepInfer: All the Cost-Saving Tricks Laid Bare
DeepSeek open-sources inference engine DeepInfer and Recipe. We break down PD disaggregation, KV cache, SWA bounded replay, speculative decoding, and what $38 unit economics really means.
DeepInfer Goes Open Source: $330K Annual Revenue Per GPU — How a Inference Framework Is Disrupting AI Compute Economics
DeepSea open-sources DeepInfer, disclosing over $330K annual revenue per GPU and gross margins up to 96.94%. We break down how prefill/decode disaggregation and KV Cache disk offloading cut inference costs by 96%, and compare revenue certainty against mining.
DeepSeek Open-Sources Inference Engine DeepInfer: The AI Cost War and the Contest Over National Destiny
DeepSeek open-sources DeepInfer under Apache 2.0, boosting prefill throughput 25x and supporting NVIDIA SM100 and Huawei Ascend 950. A deep dive into its optimizations, open-source strategy, and the AI cost war.
DeepSeek Open-Sources Inference Framework Deepinflow: The AI Price War Moves into the Back Kitchen
DeepSeek open-sources its internal inference framework Deepinflow, revealing optimization experience in high-performance operators, parallelism strategies, and speculative decoding—challenging the high-priced API model of OpenAI and other giants by open-sourcing deployment efficiency to disrupt AI pricing logic and push large model inference from luxury good to public infrastructure.
DeepSeek's Open-Source Infra Map: What Ordinary Programmers Should Understand
A visual guide to DeepSeek's open-source infrastructure: Flash MLA, DeepGEMM operators, DeepEP communication, 3FS storage, and DeepInfer inference—what each library solves, how they fit together, and what ordinary programmers can take away.
DeepSeek Open-Sources DeepInfer Inference Framework: A Revolution in Local LLM Deployment Efficiency
DeepSeek open-sources DeepInfer, its internal inference framework integrating speculative decoding, MoE async scheduling, and high-performance operators to boost local LLM deployment efficiency.
DeepSeek Open-Sources DeepInfer: How a 72-Hour Context Cache Rewrites AI Inference Costs
DeepSeek releases open-source inference framework DeepInfer, using 72-hour context caching to slash AI inference costs—up to $38 revenue per GPU per hour. We break down the tech, hardware strategy, and the open vs. closed source battle.
DeepSeek Open-Sources DeepInfer Inference Engine — Why Did It Trigger a US Ban?
DeepSeek open-sources its inference engine DeepInfer for commercial use with impressive performance, yet it was banned by US federal civilian agencies. This article analyzes DeepInfer's technical features, the significance of open sourcing, and the game behind the ban.
DeepSeek Open-Sources Inference Framework DeepInfer: The Secret to Halving AI Inference Costs
DeepSeek open-sources DeepInfer inference framework, revealing online inference engineering secrets: PD separation, KV Cache, MoE load balancing, and speculative decoding—how it earns $38/GPU/hour, avoids recomputing 96% of input, plus open-source boundaries and industry impact.
4 DGX Sparks Test DeepSeek V4.1 Flash: How Good Is Local Inference Productivity?
Four DGX Sparks in a 200G DAC ring tested running DeepSeek V4.1 Flash locally: code generation 119.8 tokens/s, 369.9 tokens/s across 16 channels, ~5 min first token at 1M context, 0.4s after cache hit. Includes four agent task tests and deployment workflow.
GLOBE: Geometry-Guided Early Stopping for Reasoning — Halving Output Without Losing Accuracy
GLOBE is a training-free online early-stopping algorithm that uses unit-sphere semantic trajectories to decide when reasoning can stop, cutting output tokens by 49.7% and latency by 51.0% across five benchmarks while matching full CoT accuracy.
Build Your Own Inference Stack: A Practical Guide to Running Production-Grade AI Agents on Dedicated GPUs
Based on an Akamai workshop recording, a detailed guide to building a production-grade AI agent inference stack on dedicated GPUs: vLLM deployment, VRAM budgeting, real-world FP8 quantization gains, speculative decoding pitfalls, and concurrency tuning.
Deepgram Voice AI Optimization: Porting Nova-3 to AWS Trainium2 in Practice
Deepgram ported Nova-3 to AWS Trainium2 using the Neuron stack, kernel-level tuning, and a Rust data-parallel runtime to boost batch inference throughput and cut costs.
One Machine Becomes Two: LLM Inference Disaggregation and the Engineering Logic Behind a 75% Throughput Gain
Explaining PD disaggregation for LLM inference: from Splitwise to Prime Inference's GB200 tuning, 75% more requests on the same GPUs, with KV transfer at 3-4% of TTFT.
vLLM Introduces Push-Based KV Transfer: 3x TTFT Improvement in Disaggregated Inference
vLLM introduces a push-based NIXL connector to optimize KV cache transfer in disaggregated inference. On Amazon EC2 P5en with AWS EFA, TTFT improves up to 3x and TPOT by 30%.
How KV Cache Reshapes AI Agent Cost Structures
Analyzing how KV Cache affects AI Agent operating costs: shrinking cache saves memory, reusing prefixes cuts redundant compute, nearby storage reduces transmission latency — each saves different resources.
vLLM P/D Disaggregation: The Calculation Rule Where KV Transfer Cost Determines Benefit
Is vLLM prefill/decode disaggregation worth it? We break down KV cache transfer bytes and bandwidth ratios, showing overhead for 70B models from 100GbE to 400Gb links, with a p99 ITL measurement script and a decision table.