#LLM inference acceleration

331 related articles

Jul 7, 2026·6 min

Claude Code Chinese Guide: A Complete Path from Zero to an Efficient Workflow

A systematic Claude Code learning guide built for Chinese developers, covering ten core modules including Slash Commands, Memory, MCP, and Hooks, with a three-tier path to build an AI coding workflow in 11–13 hours.

Jul 7, 2026·3 min

Claude Code Caching Explained: A Practical Guide to Saving Money and Speeding Up

A deep dive into Claude Code's Prompt Caching mechanism, covering the three-layer cache structure, invalidation rules, lifecycle management, and best practices to reduce API costs and boost response speed.

Jul 7, 2026·7 min

Dispersion Loss: A Key Technique for Solving Embedding Condensation in Small Language Models

Embedding condensation is a hidden bottleneck in small language model training. Dispersion Loss combats this by enforcing representation spread during training at zero inference cost.

Jul 7, 2026·6 min

AMD MI355X Running GLM5.2: Inference Costs at Half of NVIDIA Blackwell

AMD MI355X achieves 2,626 tokens/sec throughput running GLM5.2 at half the TCO of NVIDIA Blackwell. Deep analysis of the technical logic, ROCm ecosystem progress, and market implications.

Jul 7, 2026·4 min

NVFP4 Quantization in Practice: A Guide to Compressing Large Models with NVIDIA Model Optimizer

Deep dive into NVFP4 quantization: using NVIDIA Model Optimizer to compress Nemotron 3 Ultra to FP4 checkpoints, reducing memory by 75% and boosting inference throughput on Blackwell GPUs.

Jul 7, 2026·3 min

DeepSeek Open-Sources DiSpark: Up to 85% Faster AI Inference Without Changing Hardware

DeepSeek and Peking University release DiSpark, an open-source framework that speeds up LLM inference by 60–85% using speculative decoding and hierarchical verification — no hardware changes or retraining required.

The Complete Guide to Local LLM Deploy…

Jul 7, 2026·2 min

The Complete Guide to Local LLM Deployment: Tools, Hardware, and Model Selection

A complete guide to locally deploying open-source LLMs: covering VRAM requirements, quantization, tools like Ollama and LM Studio, and model selection tips for Llama, Qwen, and more.

Jul 7, 2026·5 min

Defending the Right to Run Local AI: A Battle for Privacy, Freedom, and Digital Sovereignty

Local AI faces a triple threat from tightening regulation, hardware lock-downs, and commercial pressure. A deep analysis of why running open-source LLMs on your own device is a digital right worth defending.

Jul 7, 2026·3 min

Unsloth Major Update: GLM-5.2 Support, 3x Longer Context, and a New Model Hub

Unsloth v0.1.471-beta adds full GLM-5.2 support, 3x longer context (up to 200K tokens on a single GPU), a new Model Hub, and Chat Canvas — a major leap for local LLM fine-tuning.

Jul 7, 2026·3 min

Unsloth Major Update: DiffusionGemma & Gemma 4 MTP Support, 2x Inference Speed Boost

Unsloth v0.1.464-beta adds DiffusionGemma, Gemma 4 MTP, and MiniMax-M3 support, delivering ~2x inference speed boost, new Hub, RAG Q&A, tensor parallelism, and full CUDA/ROCm/Windows coverage.

Jul 7, 2026·3 min

AMD Ryzen AI Halo Local AI Dev Kit Deep Dive: Is $4,000 Worth It?

AMD Ryzen AI Halo dev kit at $4,000 features 128GB unified memory and XDNA 2 NPU for local LLM inference. Deep dive into architecture, performance trade-offs, vs. Mac Studio, and software ecosystem challenges.

Jul 7, 2026·6 min

14x Speedup: How Manticore Rebuilt the ONNX Embedding Inference Path

Manticore Search restructured its ONNX inference path to achieve 14x faster text embeddings. Deep dive into batching, session reuse, zero-copy memory, and thread tuning for vector search systems.

Jul 7, 2026·7 min

Claude Sonnet 5 Released: The Most Capable Agent Model at Mid-Tier Pricing

Anthropic releases Claude Sonnet 5, its most agentic Sonnet model yet. With planning, browser/terminal tool use, and autonomous execution, it brings flagship Agent capabilities to mid-tier pricing.

Jul 7, 2026·7 min

Claude Sonnet 5 Released: The Most Powerful Agent-Capable Mid-Tier AI Model

Anthropic releases Claude Sonnet 5, its most agentic mid-tier model with planning, browser/terminal tool use, and autonomous execution—bringing flagship Agent capabilities at significantly lower cost.

Jul 6, 2026·5 min

Building RAG from Scratch: Creating a Domain Expert System with Local Small Models

Step-by-step guide to building a complete RAG pipeline with Ollama + LangChain + FAISS + Qwen 1.5B. Run document retrieval and intelligent Q&A locally without a GPU.

Smart Routing to Cut Claude Code Token…

Jul 6, 2026·2 min

Smart Routing to Cut Claude Code Token Costs: How It Works and How to Apply It

Learn how smart routing reduces Claude Code token costs through tiered model scheduling, context compression, and request caching — a practical guide for developers and teams.

Jul 6, 2026·2 min

AI Large Language Model Job Seeker's Guide: Engineering & Deployment vs. Algorithm Research — A Complete Breakdown

Confused about breaking into AI LLMs? This guide breaks down the two core career tracks — Engineering & Deployment vs. Algorithm Research — covering RAG, Agents, and more.

Jul 6, 2026·3 min

Running Local LLMs on AMD GPUs: Ditching Ollama for LM Studio Delivers 7x Speed Boost

AMD GPU black screens running local LLMs? This post-mortem covers Ollama's 3 fatal flaws and how switching to LM Studio boosted token speed from 5 to 36, with ROCm setup, Speculative Decoding, and GFX version tips.

Jul 6, 2026·2 min

Ollama: The 175K-Star Tool for Running Local LLMs with One Command

Ollama is an open-source local LLM runner with 175K+ GitHub Stars. Built in Go, it supports Llama, Mistral, Qwen and more — deploy in 3 steps, no setup headaches.

Jul 6, 2026·3 min

Short Drama Agent Tested: Is AI One-Click Video Production Actually Feasible?

Full hands-on test of Short Drama Agent: from scriptwriting and character three-view sheets to AI video generation. We break down the workflow for cute-style and xianxia dramas and analyze three key pain points: cost, rigidity, and visual inconsistency.