#inference speedup

113 related articles

Jul 12, 2026·3 min

CPU-Native Language Models: Breaking Through Inference Bottlenecks with Memory Bandwidth Co-Design

SiliconLLM builds a CPU-native LLM architecture from scratch, combining selective SSM, ternary (1.58-bit) LUT MLP, and granular MoE, co-designed around the L3 cache bandwidth cliff. Ternary kernels show 4-5x speedup over fp32.

Jul 12, 2026·4 min

LingBot-World 2.0 Meets SGLang: A Deep Dive into the Evolution of Real-Time Interactive World Models

SGLang-Diffusion now officially supports LingBot-World 2.0, delivering leaps in resolution and temporal consistency. With live sessions, chunked streaming, and camera control, world models achieve low-latency controllable interaction.

Jul 12, 2026·4 min

INT4 ConvRot W4A4 Quantization: Running Large Image Models on Low-VRAM GPUs in ComfyUI

An in-depth look at INT4 ConvRot W4A4 quantization, covering conversions of Krea2, Qwen-Image, and other diffusion models to help ComfyUI users run large image models on 8GB GPUs.

Jul 12, 2026·5 min

GPT-5.6 Released: Hands-On Analysis of Programmatic Tool Calling and Autonomous Agents

GPT-5.6 is officially released with core upgrades including programmatic tool calling, autonomous subagent delegation, and higher token information density. A hands-on card game build reveals its Agentic power.

Jul 11, 2026·5 min

GPT-5.6 Triple Launch: Understanding the Positioning and Key Differences of Sol, Terra, and Luna

OpenAI released three GPT-5.6 models—Sol, Terra, and Luna—covering everything from flagship reasoning to lightweight speed. A deep dive into their positioning, performance differences, pricing, and industry signals.

Jul 11, 2026·7 min

Practical RL Compute Optimization: Core Strategies to Boost CPU and GPU Utilization

An in-depth look at why CPU and GPU utilization is low in RL training, covering vectorized environment parallelism, distributed Actor-Learner architectures, GPU-side simulation (Isaac Gym/Brax), and Ray RLlib practice.

Jul 11, 2026·3 min

Vibecoding in Practice: Task Concurrency Splitting Makes AI Inference 3x Faster

Learn how to split AI reasoning tasks by act and character, run 14 concurrent streams, and cut processing time from 100s to 30s — a reusable schedule-concurrency-aggregate methodology.

Jul 11, 2026·2 min

Deep Dive into Tencent Hunyuan Hy3: How a 295B MoE Model Cracks the Agent Deployment Challenge

Tencent Hunyuan Hy3 launches with a 295B MoE architecture activating just 21B params and 256K context. Hallucination cut from 12.5% to 5.4%, MRCR nearly doubled, with MTP and EAGLE decoding and Day-0 SGLang support.

Jul 11, 2026·2 min

AI Agents for SGLang Kernel Optimization: Real-World Performance Gains and Engineering Methodology

SGLang's team converted expert knowledge into agent skills, achieving 71.4% throughput gains, TTFT reduced from 456ms to 168ms. A deep dive into agent-assisted kernel optimization methodology.

Jul 11, 2026·6 min

GPT-5.6 Now Widely Available: The Era of Three-Tier Models and Multi-Agent Orchestration Arrives

OpenAI releases GPT-5.6 (SOUL/TERRA/LUNA), with Ultra mode running four agents in parallel; Meta launches Muse Spark 1.1 with million-token context; ChatGPT desktop unifies Chat, Work, and Codex.

Jul 11, 2026·2 min

DeepSeek Speculative Decoding Algorithm Merged into vLLM Main Branch — Single-User Token Speed Jumps Dramatically

DeepSeek's speculative decoding algorithm (DSpark) is now merged into vLLM main branch, natively supporting Qwen3 and Gemma. Tests show ~150× single-user token speed gains and ~40–50% throughput improvement.

Jul 10, 2026·4 min

DeepSeek-V4 Deep Dive: Million-Token Context Architecture Breakthrough and Core Innovations

Deep dive into DeepSeek-V4: 1.6T-parameter MoE, CSA+HCA hybrid attention, MHC & MUON optimizer. Inference FLOPs drop to 27% of V3.2, redefining open-source LLM SOTA.

Jul 10, 2026·5 min

DeepSeek Boosts Inference Speed by 85%: How Speculative Decoding Is Reshaping the AI Speed Race

DeepSeek and Peking University's DS Spark paper boosts AI inference speed by up to 85% via confidence scheduling and semi-autoregressive speculative decoding — no model or GPU changes.

Jul 10, 2026·6 min

AI's Second Half: The Decisive Battle Shifts from Models to Infrastructure

Model capabilities are converging, making inference cost and scalability the new focus of AI competition. A deep analysis of AI infrastructure's core layers.

Jul 10, 2026·7 min

Reverse-Engineering CUDA-checkpoint: The Technical Principles and Practice of Cracking GPU Cold Starts

An in-depth analysis of reverse-engineering Nvidia CUDA-checkpoint to accelerate GPU cold starts. Covers checkpoint/restore, Serverless GPU prospects, and VRAM snapshot challenges.

Jul 9, 2026·6 min

Unsloth Major Update: DeepSeek-V4 Support and a Deep Dive into NVFP4 Quantized Export

Unsloth v0.1.481-beta adds full DeepSeek-V4-Flash support, NVFP4/FP8/imatrix GGUF quantized export, 1.3x faster GRPO, 3-5x faster MoE training, and an OpenAI-compatible API service in Studio.

Jul 9, 2026·5 min

DeepSeek DSpec Hands-On: How Speculative Decoding Accelerates LLM Inference by 6x

Hands-on report on DeepSeek's open-source inference acceleration toolkit DSpec: draft model + smart scheduling delivers lossless speedup, hitting acceptance length 6 on GSM8K and reproducing official data.

Jul 9, 2026·7 min

Block-Sparse Featurizers: Opening the Black Box of Vision Model Neural Geometry

Block-sparse featurizers remap dense vision model activations into block-sparse representations, making the internal feature spaces of ViT, CNN, and other models readable and interpretable. This article explores their core principles, links to mechanistic interpretability, and applications.

Jul 8, 2026·6 min

Made by Google Event Confirmed: New Pixel Phones with AI Capabilities Coming Soon

Google confirms the Made by Google hardware event in NYC, unveiling next-gen Pixel phones with new Tensor chips and deep Gemini AI integration. On-device AI gets a major upgrade.

Jul 8, 2026·7 min

Electron + LangGraph in Practice: Building an AI Agent Visual Orchestration App from Scratch

A hands-on guide to building an enterprise-grade AI Agent workflow orchestration app with Electron Forge and LangGraph, covering local LLM deployment (Qwen3-0.6B), node-based visual canvas design, and full Function Calling integration.