#AI inference optimization
1877 related articles
Industry Insights·3 min
Should Custom Software Teams Go All-In on LLMs? ChatGPT Itself Gave the Answer
Should custom software teams go all-in on LLMs? Through a deep conversation with ChatGPT, we analyze scenario selection, Token costs, and industry fit to provide a rational decision framework for ToB developers.
Read more →
Product Reviews·2 min
Agent Memory: Giving AI Coding Agents Persistent Memory Across Sessions
Agent Memory is an open-source local memory layer providing persistent, cross-session, cross-tool long-term memory for AI coding agents like Claude Code, Cursor, and Codex.
Read more →
Tech Frontiers·2 min
Claude Opus 4.8 Thinking Effort Calibration Explained: A Critical Optimization Direction for AI Reasoning Models
Anthropic releases Claude Opus 4.8 with optimized thinking effort calibration. This article explains what it is, why it matters for AI reasoning models, and its impact on industry competition.
Read more →
Product Reviews·3 min
Codex Computer Use Hands-On Review: Setup Guide, How It Works, and Security Risks Explained
Hands-on review of OpenAI Codex Computer Use: step-by-step setup guide, WeChat demo, screenshot-based core principles, and analysis of three major security risks including privacy leakage and current limitations.
Read more →
Product Reviews·2 min
Claude Opus 4.8 Real-World Testing: What $50 in Tokens Actually Gets You
In-depth testing of Claude Opus 4.8 across game dev, UI reproduction, 3D scenes, and tool building—$50 in Tokens reveals its true capabilities and limits.
Read more →
Tech Frontiers·1 min
Windsurf Integrates Claude Opus 4.7 Fast Mode with 2.5x Speed Boost
Windsurf integrates Claude Opus 4.7 fast mode with 2.5x speed boost while retaining full intelligence. Analysis of its impact on developer productivity and AI coding tool competition.
Read more →
Research·2 min
Agent Loops in Practice: Transforming Token Output into Productivity from CUDA Kernels to Automated Research
Deep dive into how the Humanize framework transforms LLM tokens into engineering productivity via Agent Loops. Covers KDA winning CUDA kernel contests, virtual hardware optimization, and 50% research cost reduction.
Read more →
Tutorials·2 min
Tutorial: Deploying a PD-Disaggregated SGLang Multi-Node Inference Cluster on AMD GPUs
Learn how to deploy a PD-disaggregated SGLang inference cluster on AMD GPUs using a single config file, boosting LLM throughput and latency performance.
Read more →
Tech Frontiers·2 min
SGLang v0.5.12.post1 Released: DeepSeek V4 Stability Fixes and Blackwell Adaptation
SGLang v0.5.12.post1 stability patch details: 12 critical fixes covering DeepSeek V4 garbled text and crashes, NIXL PD disaggregated inference logic, Blackwell B300 adaptation, and cold start optimization.
Read more →
Tech Frontiers·2 min
Step 3.7 Flash: Deep Dive into the 198B Sparse MoE Multimodal Model
Deep dive into StepFun AI's Step 3.7 Flash, a 198B sparse MoE vision-language model with 256K context and 3-level reasoning, excelling in multimodal understanding, AI coding, and Agent tool orchestration.
Read more →
Tech Frontiers·2 min
LFM2.5-8B-A1B: A MoE Model with 1.5B Active Parameters Delivering 4x Its Weight Class Performance
Liquid AI releases LFM2.5-8B-A1B, a MoE model with 8B total params but only 1.5B active, matching 6B-class models in tool calling. Supports 128K context, local deployment, multilingual, with SGLang Day-0 support.
Read more →
Industry Insights·2 min
SGLang Enters Finance: How AI Inference Infrastructure Is Reshaping Wall Street
SGLang co-hosts a finance AI inference event with Crusoe AI and Cloudflare, exploring LLM inference deployment in trading, risk management, and compliance — signaling Wall Street's shift to production-grade AI infrastructure.
Read more →
Industry Insights·2 min
AMD MI355X Beats B200: Full-Stack Optimization Breakdown for 5% Lower TCO on DeepSeek-R1 Inference
AMD Instinct MI355X achieves 5% lower TCO than NVIDIA B200 on DeepSeek-R1 disaggregated inference via SGLang+MoRI full-stack optimization with 1.25x per-GPU throughput.
Read more →
Tech Frontiers·1 min
Cloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang
Cloudflare contributes decode KV cache offload and Mooncake recovery fixes to SGLang, resolving garbled output under high concurrency for Kimi K2.6 and enabling automatic fault recovery in distributed inference.
Read more →
Tech Frontiers·1 min
SGLang Hosts Agent Loops Office Hour, Focusing on Agentic Loop Architecture Optimization
SGLang team hosts an Agent Loops Office Hour exploring inference optimization for agentic loops, covering KV Cache reuse, low-latency multi-turn dialogue, and tool calling techniques.
Read more →
Product Reviews·3 min
Deep Comparison of o1, o1 pro, and o3-mini-high Coding Capabilities: A Deep Research Analysis
Deep Research comparison of OpenAI o1, o1 pro, and o3-mini-high coding capabilities, covering code quality, optimization, error rates, and debugging with benchmarks and real-world cases.
Read more →
Product Reviews·3 min
Llama 3.3 70B In-Depth Review: Testing the Strongest Open-Source LLM with 13 Questions
Meta releases Llama 3.3 70B open-source model with just 70B parameters rivaling 405B performance. Tested on 13 logic, math, and coding questions, it passed 12 — reshaping the open-source model landscape.
Read more →
Industry Insights·3 min
Six Foundational Upgrades to Claude Code: AI Programming Moves from Lab to Industrial Scale
Anthropic's largest-ever foundational upgrade to Claude Code fixes six critical issues at once—terminal flickering, thinking freezes, cryptic errors, context deadlocks, unstable connections, and session crashes—shifting AI coding competition to the infrastructure layer.
Read more →
Industry Insights·3 min
Deep Dive into Three Major LLM Career Paths: Requirements, Tech Stacks, and Career Prospects
Deep analysis of three core LLM roles—Application Engineer, Development Engineer, and Algorithm Engineer—covering technical requirements, salary thresholds, and career prospects including RAG, fine-tuning, and inference deployment.
Read more →
Tutorials·2 min
Zen MCP: An Open-Source Tool That Lets Claude Orchestrate Multiple AI Models in Collaboration
Deep dive into Zen MCP, an open-source project that lets Claude Code orchestrate Gemini, O3, and other AI models via MCP protocol, with cost-reduction proxy setup guide.
Read more →