#LLM inference speedup

51 related articles

Jul 25, 2026·3 min

D-Flash: The Secret Behind 3.5x LLM Inference Speedup with Diffusion Drafting

D-Flash uses fast diffusion parallel drafting and target feature KV injection to solve the autoregressive Drafter latency bottleneck in speculative decoding. 16 tokens in just 6ms, up to 3.5x speedup on HumanEval, surpassing EAGLE3 and MTP.

Jul 16, 2026·3 min

DeepSeek Open-Sources DSpark: 50%–400% LLM Inference Speedup Without Retraining

DeepSeek open-sources DSpark, delivering 50%–400% LLM inference speedup with no retraining or quantization, via semi-autoregressive drafting and confidence-scheduled verification.

Jul 27, 2026·3 min

Kimi K3 Launches on Ollama Cloud: Free Tier, User Experience, and Developer Guide

Kimi K3 officially launches on Ollama Cloud as an "extra high usage" model. This guide covers free tier quotas, cloud inference experience, technical advantages, and how developers can seamlessly call this high-performance LLM.

Jul 25, 2026·2 min

DSpark: DeepSeek and Peking University Open-Source Inference Acceleration Tech with Up to 85% Speed Boost

DeepSeek and Peking University open-source DSpark, an inference acceleration tech boosting single-user generation speed by 57%-85% under high concurrency. Learn its 3 core designs and the DSpec framework.

Jul 25, 2026·2 min

D-Flash: How Diffusion Drafting Delivers 3.5x Faster LLM Inference

D-Flash solves the autoregressive drafter latency bottleneck in speculative decoding via fast diffusion parallel drafting and target feature KV injection. 16 tokens in just 6ms, up to 3.5x speedup on HumanEval, beating EAGLE3 and MTP.

Jul 25, 2026·3 min

DeepSeek Open-Sources DeepSpec: A Full-Stack Speculative Decoding Library with 60–85% Inference Speedup

DeepSeek open-sources DeepSpec, a full-stack speculative decoding library delivering 60–85% inference speedup and up to 661% throughput gains in production. MIT license, 9 checkpoints, 9 benchmarks.

Jul 22, 2026·4 min

DeepSeek Open-Sources DeepSpec: A Full-Stack Breakdown of Speculative Decoding

DeepSeek open-sources DeepSpec, a full speculative decoding training and evaluation toolkit featuring three draft model algorithms (Ego3, DeepFlash, DeepSpark), 12 checkpoints, MIT license, and 60–85% real-world speedup.

Jul 22, 2026·3 min

V100 GPU Showdown: 4×16G PCIe vs. 2×32G SXM — Which Gives You More Bang for Your Buck?

Benchmarking 4×V100 16G PCIe vs. 2×V100 32G SXM adapter for local LLM inference. Prefill speed, decode speed, power limits, and bandwidth bottlenecks analyzed.

Block Low-Rank Compression: A Guide to…

Jul 19, 2026·2 min

Block Low-Rank Compression: A Guide to Accelerating Large Model Inference on Memory-Constrained GPUs

Learn how Block Low-Rank (BLR) decomposition compresses large model memory usage and accelerates GPU inference, including CUDA kernel optimization and combination with quantization and pruning.

Jul 17, 2026·4 min

DSpark Open Source: DeepSeek's Inference Acceleration Tech That Doubles LLM Speed

DeepSeek and Peking University open-source DSpark, an inference acceleration technology using semi-autoregressive architecture and dynamic scheduling to boost LLM speed by 50%+ and double GPU concurrency without quality loss.

Jul 16, 2026·3 min

AI Daily: Alibaba Open-Sources Dance Model, World Model Launches, First AI-Native Phone Debuts

Alibaba open-sources 14B dance model Wan-Dancer, AutoNavi launches World Studio, Stepfun debuts AI-native phone STEPS NEO; GPT-5.6 file deletion and AI companion shutdowns spark safety and regulation debates.

Jul 15, 2026·4 min

Running Qwen3.6 27B Locally on a Single RTX 3090: A Complete Guide to Cloud-Rivaling Performance

One used RTX 3090, one 16.8GB GGUF file, and Qwen3.6 27B runs locally offline. SWE-bench score of 77 rivals Claude Sonnet. MTP boosts speed to 59 tok/s. Full local AI coding assistant deployment guide.

Jul 15, 2026·4 min

RAM: First-Principles Approach to 50x Faster RL Post-Training for Diffusion Models

RAM (Reinforce Adjoint Matching) achieves 50x faster RL post-training for diffusion models by discarding path costs, combining ODE sampling with decorrelated training objectives. A deep dive into RAM's core principles and experiments vs. Flow-GRPO.

MemStitch Zero-Copy Context Bridging: …

Jul 15, 2026·2 min

MemStitch Zero-Copy Context Bridging: How It Achieves 25x TTFT Speedup in vLLM

A deep dive into how MemStitch's zero-copy context bridging achieves 25x TTFT speedup in vLLM. Covers KV Cache optimization, prefill acceleration, and practical developer value.

Jul 14, 2026·3 min

31 Companies Sign Agent Privacy Pact: Industry Self-Regulation Framework Goes into Effect

31 companies sign China's first AI agent privacy pact covering screen-reading authorization, training data restrictions, and payment caps. Plus: domestic LLM tops OpenRouter, Meta's $40B compute bet, and agent commercialization challenges.

Jul 13, 2026·4 min

AI Weekly Recap: Four Major Vendors Cut Prices Together, Copilot's Paid Conversion Rate Just 4.5%

In one week, OpenAI, xAI, Google, and Microsoft all cut AI prices, driving near-frontier inference costs sharply lower. Meanwhile, Microsoft Copilot's paid conversion across 450M seats is under 4.5%, exposing the monetization challenge of general AI assistants.

Jul 12, 2026·2 min

LingBot-Video 1.3B Selective FP8 Quantization Benchmark: 22% Faster Sampling

Reddit developer ALX-CODE shares a selective FP8 quantization scheme for LingBot-Video 1.3B, achieving ~22% faster sampling (4.65s→3.65s) on an RTX 5080. This article breaks down the mixed-precision strategy, open-source resources, and ComfyUI adaptation.

Jul 12, 2026·4 min

Unsloth Releases Qwen3.6 NVFP4 Quantization: 2.5x Faster with Zero Accuracy Loss

Unsloth releases NVFP4 quantization for Qwen3.6 using W4A4 true 4-bit Tensor Core computation, delivering up to 2.5x inference speedup over NVIDIA's official implementation with accuracy matching or exceeding BF16 on benchmarks like MMLU-Pro.

Jul 12, 2026·3 min

Benchmarking LLMs on Smartphones: A Deep Dive into the MELTing Point Benchmark

The MELTing Point paper is the first to evaluate mobile LLM performance in real user scenarios, covering iPhone, Samsung, Pixel and more, testing TinyLlama, Mistral-7B and others—revealing GPU inference gains, 47°C heat warnings, and prefill-decode disaggregation.

Jul 12, 2026·5 min

GPT-5.6 In-Depth Field Test: A Real-World Log After Burning Through $200K

A developer stress-tested GPT-5.6 for six weeks across 67 projects, burning $180K-$240K in inference. Real cases of task persistence, Rust rewrites, autonomous browser control — plus honest frontend and 3D shortfalls.