#推理优化
61 related articles
New Breakthrough in Peak Memory Scheduling: Exact Implementation and Provable Optimality
This article analyzes the latest research on peak memory scheduling: achieving reliable peak memory reduction through exact modeling and statement permutation, providing provable optimality certificates, and exploring the theoretical boundaries of selective recomputation, offering new ideas for edge AI inference optimization.
DeepSeek V4.1 Flash 1017 Technical Breakdown: How the MoE Architecture Breaks Through the Throughput Bottleneck
A deep technical breakdown of DeepSeek V4.1 Flash 1017 (Max): how MoE dynamic routing solves expert starvation, how MLA compresses the KV cache to break the memory wall, and the kernel-level tuning and deployment constraints behind TTFT and throughput gains.
Veloce-Engine: The Technical Inside Story of Real-Time AI Inference Orchestration
A deep dive into the Veloce-Engine real-time AI inference orchestration layer: adaptive routing, semantic caching that cuts redundant calls by 70%, rate-limit retries, and streaming responses. Learn about its architecture, use cases, and pricing.
GPUStack vLLM Production Deployment: KV Cache Pinning and OOM Avoidance in Practice
Real-world ops lessons from a four-node mixed GPU cluster: pinning KV Cache in GPUStack + vLLM production deployments, avoiding 70B model OOM, and diagnosing TTFT and RSS decay.
DuckDB and Arrow Dominate AI Data Preparation—Why Do They Hit a Wall in Inference Serving?
Columnar engines like DuckDB and Arrow excel at AI data preparation but hit bottlenecks in inference serving. This article analyzes the structural reasons and presents an architecture that separates offline columnar storage from online row-based storage, emphasizing Arrow's true value as a zero-copy exchange format.
Implementing GPT-2 KV Cache from Scratch: Correctness, Latency, and Memory Benchmarks
Implementing GPT-2 small KV Cache from scratch with benchmarks: 6.1e-5 logit error, only 6x single-token speedup at context 1000 (not 1000x), 72KiB/token formula plus 8.5MB cuBLAS workspace offset, and a multi-token decoding failure case.
llama.cpp MoE Cache Benchmark: Doubling 30B Model Decode Speed on a T4
llama.cpp's new MoE expert cache doubles Qwen3-30B-A3B decode speed on a free Tesla T4, from 22.6 to 43.2 tokens/s. Cache mechanics, config script, benchmarks, and concurrency pitfalls.
NVFP4 vs FP8 vs GGUF: 2.3x Faster Decoding on 16GB Ada GPUs
Same model Nemotron Dory 1.0 4B compared on RTX 4070 Ti SUPER across NVFP4, FP8, MXFP8, GGUF, and BF16: NVFP4 decodes 2.3x faster, matches BF16 on GSM8K, FP8 leads long-prompt prefill, MXFP8 best token-level fidelity.
Full-Stack Optimization for LLM Inference: GPU, CPU, and Frontend Acceleration
Explore full-stack optimization for LLM inference: GPU-layer wide-EP with EPLB for load balancing, CPU All-Reduce tuning, and NVIDIA Dynamo with a Rust frontend for KV-aware routing.
AMD MI355X Beats B300 in Inference Performance: 24.4% Higher Throughput on 8 GPUs
AMD MI355X shows inference strength in SemiAnalysis AgentX: 7,567 tok/s per GPU on 8 GPUs, 24.4% above NVIDIA HGX B300 and 87.4% more tokens per dollar. We break down the vLLM optimizations and AMD kernels that narrowed the gap.
Cloud T4 Falls Short of Spec: Gemma 4 Decoding Slows 7% Due to Heat
Measured thermal drift on a single Google Cloud Tesla T4 with Gemma 4 E2B inference: 115.3 tokens/s decoding at a cold 45°C, falling to 106.8 at a hot 77°C — a sustained 7.4% slowdown. Thermal throttling never triggers; the power cap and leakage current are the real culprits.
SSR Framework: Liberating Small-Model Multimodal Search Agents from Lengthy Reasoning
SSR framework proposes a selection-over-generation multimodal reasoning paradigm, freeing small-model search agents from lengthy ineffective reasoning and dramatically cutting inference costs for resource-constrained scenarios.
Delta Tokenizer: How Session Deltas and Multi-Core Parallelism Speed Up Long-Prompt Tokenization by 40x
Delta Tokenizer is a CPU tokenization frontend that uses session-delta encoding and multi-core segmented parallelism to speed up single-turn tokenization by ~40x, cutting median TTFT by 59% at 100K tokens in SGLang and P99 TTFT by 9.5x under load.
GeoThinker-F: A New Framework That Frees Multimodal LLMs from External 3D Encoders for Spatial Intelligence
GeoThinker-F proposes an internally predicted world representation framework that frees multimodal LLMs from external 3D geometry encoders through two-stage decoupled training, achieving 3.6x faster inference and reducing latency from 1.45s to 0.40s while maintaining spatial reasoning performance.
DoubtLess: Training-Free Contrastive Decoding That Helps Reasoning Models Think Less and Answer Right
DoubtLess is a training-free contrastive decoding method that uses a small reasoning model to guide large models toward shorter chain-of-thought. On seven benchmarks, it cuts Qwen3-32B's CoT by 31% with unchanged accuracy, reduces thinking tokens per turn by 20% in coding agent scenarios, and generalizes across model families.
DeepSeek V4.1 Flash: How Read-Write Separation Reshapes Long-Conversation Costs
DeepSeek V4.1 Flash uses read-write separation—shorter input paths, shared attention memory, and selective reconstruction—to restructure inference costs in long conversations.
Inside the LLMD Project: How Open Source Inference Achieves Production-Grade Scale
IBM Distinguished Engineer Carlos Costa shares LLMD project updates: how open source inference achieves production-scale with agentic serving performance, multi-tenant QoS, and zero-downtime ops, to be showcased with Red Hat and Google at PyTorch Conference.
Multi-Node LLM Inference Orchestration Topologies: Performance Bottlenecks and Architectural Trade-offs
A deep dive into multi-node LLM inference orchestration topology evaluation: the synchronization latency of cross-node tensor parallelism over standard Ethernet, and the predictability advantages and architectural trade-offs of disaggregated prefill/decode with KV-aware routing in high-concurrency multi-turn scenarios.
From Silicon to Token Economics: Full-Stack Optimization of AI Inference Infrastructure
Dylan Patel of SemiAnalysis joins Fireworks Forge to discuss full-stack AI inference optimization and token economics, from chip architecture to the application layer.
SGLang-Diffusion: A High-Performance Serving Framework for Diffusion Models
SGLang-Diffusion is a high-performance inference engine and serving stack for diffusion models, supporting offline and real-time workloads with parallelization and execution path optimizations. Debuting at PyTorch Conference 2026.