#TTFT optimization
38 related articles
·3 min
TB/s Memory Tier Accelerates SGLang: How X-Mem Reduces TTFT by 6.7x
Netpreme integrates X-Mem™ MPU into SGLang HiCache, achieving up to 6.7× TTFT reduction and 33–50% TPS gains at 98% prefix cache hit rates. Here's the technical breakdown.
Read more →
·2 min
AI Agents for SGLang Kernel Optimization: Real-World Performance Gains and Engineering Methodology
SGLang's team converted expert knowledge into agent skills, achieving 71.4% throughput gains, TTFT reduced from 456ms to 168ms. A deep dive into agent-assisted kernel optimization methodology.
Read more →
·5 min
Real-World Comparison of 6 Open-Source LLMs for On-Premise Deployment: A Selection Guide to Avoid Pitfalls
A hands-on comparison of 6 open-source LLMs (DeepSeek, Qwen3, Zhipu GLM, Kimi K2, MiniMax M3, Tencent Hunyuan 3) for on-premise deployment—covering hardware cost, inference efficiency, and deployment difficulty.
Read more →
·6 min
Tencent Hunyuan HY3 Official Version Open-Sourced: The Strongest Under 300B, at 1 Yuan per Million Tokens
Tencent Hunyuan HY3 official version is open-sourced under Apache 2.0, priced as low as 1 yuan per million input tokens, with major gains in agents, reasoning, coding, and long context. On the same day, Meituan open-sourced its trillion-parameter LongCat 2.0.
Read more →
·6 min
Unsloth Major Update: DeepSeek-V4 Support and a Deep Dive into NVFP4 Quantized Export
Unsloth v0.1.481-beta adds full DeepSeek-V4-Flash support, NVFP4/FP8/imatrix GGUF quantized export, 1.3x faster GRPO, 3-5x faster MoE training, and an OpenAI-compatible API service in Studio.
Read more →
·2 min
GitHub Copilot Integrates Claude Opus 4.8 Fast Mode: AI Coding Tools Are Now Competing on Response Speed
GitHub Copilot adds Claude Opus 4.8 Fast Mode to preview. Learn how it boosts Token speed for interactive coding and agent workflows, plus pricing and enterprise management details.
Read more →
·7 min
AI Agent Development: A Four-Stage Competency Model from 15K to 40K RMB Salary Growth Path
A detailed four-stage competency model for AI Agent development: from Python/RAG basics (15K) to workflow orchestration (20K), inference optimization (30K), and Agent cluster governance (40K RMB).
Read more →
Has AI Coding Assistant Fable Been Ner…
·4 min
Has AI Coding Assistant Fable Been Nerfed? Developer's Routing Anomaly Report Sparks Debate
A developer questioned whether AI coding assistant Fable was nerfed, finding 4–10x more tokens routed to Opus with Fable doing ~20% of the work. A deep dive into multi-model routing, transparency, and AI trust.
Read more →
·2 min
MiniMax M3 Launches on Fireworks: 512K Context and MSA Sparse Attention Explained
MiniMax M3 launches on Fireworks with 512K context and multimodal input. MSA sparse attention delivers 9x prefill and 15x decode speedups. Deep dive into architecture, pricing, and open-model competition.
Read more →
·3 min
Fireworks AI Launches Qwen 3.7 Plus: Zero Data Retention and 99.9% SLA for Enterprise Deployment
Fireworks AI launches Qwen 3.7 Plus with latency/throughput optimization, zero data retention, and 99.9% SLA enterprise guarantees. Explore the full-stack deployment solution for commercial open-source model inference.
Read more →
·2 min
A Complete Guide to LLM Infrastructure: Core Challenges from GPU Clusters to Inference Optimization
A deep dive into core challenges and key technologies for LLM infrastructure, covering GPU cluster management, inference optimization, distributed training, cost control, and observability.
Read more →
·2 min
Model Router Prism Integrates Fable 5: 30% Cost Reduction Without Quality Loss
AI model router Prism announces Fable 5 integration, achieving up to 30% cost reduction per task without quality loss through per-turn intelligent routing and cache-aware technology.
Read more →
Industry Insights·3 min
AI API Aggregation Gateways: A Complete Analysis of Pros and Cons for Unified Multi-Model Access
In-depth analysis of API aggregation gateways for multi-model AI access: unified interfaces, intelligent routing, disaster recovery, plus key risks around security, latency, and compliance.
Read more →
Tech Frontiers·1 min
Windsurf Integrates Claude Opus 4.7 Fast Mode with 2.5x Speed Boost
Windsurf integrates Claude Opus 4.7 fast mode with 2.5x speed boost while retaining full intelligence. Analysis of its impact on developer productivity and AI coding tool competition.
Read more →
Tutorials·2 min
Tutorial: Deploying a PD-Disaggregated SGLang Multi-Node Inference Cluster on AMD GPUs
Learn how to deploy a PD-disaggregated SGLang inference cluster on AMD GPUs using a single config file, boosting LLM throughput and latency performance.
Read more →
Tech Frontiers·2 min
SGLang v0.5.12.post1 Released: DeepSeek V4 Stability Fixes and Blackwell Adaptation
SGLang v0.5.12.post1 stability patch details: 12 critical fixes covering DeepSeek V4 garbled text and crashes, NIXL PD disaggregated inference logic, Blackwell B300 adaptation, and cold start optimization.
Read more →
Deep Dives·2 min
Claude Opus vs. Sonnet vs. Haiku: How to Choose the Right Model
Compare Anthropic's Claude Opus, Sonnet, and Haiku models across intelligence, speed, and cost. Practical selection guide with multi-model routing strategies.
Read more →
Deep Dives·2 min
How Fast Is 10 Tokens Per Second from an LLM? An Open-Source Tool for Visualizing Token Generation Speed
An open-source tool lets you visualize LLM token generation speeds from 5-800 TPS, helping developers make rational choices about model selection and inference optimization.
Read more →