#KV Cache Offload

42 related articles

Jul 12, 2026·4 min

AMD Ryzen AI Halo Hands-On: Can x86 Architecture Challenge NVIDIA's DGX Spark?

In-depth review of the AMD Ryzen AI Halo mini AI box: powered by the Ryzen AI Max Plus 395 (Strix Halo) chip with 128GB unified memory, priced at $4,000. Compared against NVIDIA's DGX Spark across token generation, prefill speed, and x86 advantages.

Jul 11, 2026·3 min

TB/s Memory Tier Accelerates SGLang: How X-Mem Reduces TTFT by 6.7x

Netpreme integrates X-Mem™ MPU into SGLang HiCache, achieving up to 6.7× TTFT reduction and 33–50% TPS gains at 98% prefix cache hit rates. Here's the technical breakdown.

Jul 11, 2026·3 min

Connecting Codex to Ollama: A Complete Guide to Deploying AI Large Models Locally

A step-by-step guide to combining Codex with Ollama to deploy open-source AI large models locally. Private data, no subscription, offline operation, no VPN needed. Includes hardware selection and setup.

Jul 11, 2026·6 min

Why Is Qwen3 27B So Slow on a 4090? A Hands-On Guide to Local Inference Performance Optimization

RTX 4090 taking over 400 seconds to run Qwen3 27B inference? This article analyzes the core causes—VRAM overflow and CPU offloading drag—and offers targeted fixes.

Jul 10, 2026·5 min

TensorRT Multi-Device Inference: Engineering Practices for Breaking Through Single-GPU Memory Bottlenecks

NVIDIA TensorRT now supports multi-device inference via pipeline and tensor parallelism, distributing large models across multiple GPUs to break through single-card memory limits.

Jul 10, 2026·6 min

Samsung's Chip Profit in a Single Year Exceeds 40 Years Combined: The AI Wave Triggers a Memory Price Crisis

Samsung's chip division is set to earn more profit in one year than in its past 40 years combined, with quarterly profit surging 19x YoY, surpassing NVIDIA. AI data centers are seizing HBM and DRAM capacity, driving up DDR5 and SSD prices.

Jul 10, 2026·3 min

Running LLMs Locally: The Illusion vs. the Real Value of Open-Weight Models

Open weight ≠ runnable locally. This article breaks down the hardware barriers, VRAM limits, electricity costs, and parallelism constraints of models like GLM 5.2 and DeepSeek — revealing where open-weight models truly add value: driving cloud competition, not home replication.

Jul 9, 2026·6 min

Rising RAM Prices Push Up Gaming Laptop Costs? A Value-Focused Buying Guide

Rising RAM and storage prices have eroded gaming laptop value. This guide breaks down the causes, cost pass-through, and offers practical buying advice plus MSI value picks.

Jul 8, 2026·7 min

Ollama + Hermes Local AI Deployment: Building a 100% Private Personal AI Operating System

Build a fully private local AI system with Ollama + Hermes: zero cost, no rate limits, data stays local. Learn deployment steps, model selection tips, and private/cloud hybrid workflows.

Jun 14, 2026·3 min

Complete Guide to Alibaba Cloud Website Architecture: The Full Request Path from DNS to Auto Scaling

A systematic guide to Alibaba Cloud website architecture covering DNS, CDN, WAF, CLB/ALB, ECS, Redis, NAS/OSS, and auto scaling along the full user request path.

Jun 14, 2026·3 min

5 Ways to Deploy LLMs Locally: From Getting Started to Production

A complete guide to 5 local LLM deployment methods: LlamaCPP, Ollama, LM Studio, vLLM/SGLang, and MLX-LM — from personal dev to production environments.

Jun 13, 2026·2 min

A Complete Guide to LLM Infrastructure: Core Challenges from GPU Clusters to Inference Optimization

A deep dive into core challenges and key technologies for LLM infrastructure, covering GPU cluster management, inference optimization, distributed training, cost control, and observability.

Jun 7, 2026·3 min

Hands-On Testing of DS4 Engine by Redis Creator: How Does DeepSeek V4 Perform Locally on a 128GB Mac?

Redis creator Antirez's DS4 inference engine tested: running DeepSeek V4 Flash locally on a 128GB Mac via asymmetric structure-aware quantization, with real-world coding benchmarks.

llama.cpp MTP Acceleration Deployment Guide: Configuration Steps & Real-World Benchmarks

Tutorials

Jun 2, 2026·3 min

llama.cpp MTP Acceleration Deployment Guide: Configuration Steps & Real-World Benchmarks

Guide to enabling MTP multi-Token prediction acceleration in llama.cpp, covering CUDA setup, desktop configuration, model selection, and benchmarks showing ~60 Token/s with Qwen3 27B.

Qoder's Context Engineering in Practice: Four-Layer Retrieval Engine and Memory System Architecture

Industry Insights

Jun 1, 2026·4 min

Qoder's Context Engineering in Practice: Four-Layer Retrieval Engine and Memory System Architecture

Deep analysis of Qoder's (Tongyi Lingma international edition) context engineering architecture, including its four-layer retrieval engine, memory engine, context caching, and core product design.

Cloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang

Tech Frontiers

May 30, 2026·1 min

Cloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang

Cloudflare contributes decode KV cache offload and Mooncake recovery fixes to SGLang, resolving garbled output under high concurrency for Kimi K2.6 and enabling automatic fault recovery in distributed inference.

Local Deployment of Qwen 3.6 27B on 4×3080Ti: Real-World Coding Test with OpenCode

Product Reviews

May 27, 2026·3 min

Local Deployment of Qwen 3.6 27B on 4×3080Ti: Real-World Coding Test with OpenCode

Real-world test of Qwen 3.6 27B FP8 deployed on 4×3080Ti 16GB modded GPUs with OpenCode for system tool development. Covers hardware setup, inference speed, context management, and productivity gains.

Decoding LLM Naming Conventions: Parameter Counts, Quantization Formats & VRAM Requirements Quick Reference

Tutorials

May 27, 2026·3 min

Decoding LLM Naming Conventions: Parameter Counts, Quantization Formats & VRAM Requirements Quick Reference

Decode LLM naming conventions, understand 32B parameters & AWQ/GGUF quantization formats, with 4-bit VRAM estimation formulas, MOE model pitfalls, and model selection by GPU tier.

How to Tell If Your GPU Is Out of VRAM When Running Local LLMs: A Complete Guide to VRAM Monitoring

Tutorials

May 21, 2026·2 min

How to Tell If Your GPU Is Out of VRAM When Running Local LLMs: A Complete Guide to VRAM Monitoring

How to tell if your GPU is out of VRAM when running local LLMs. Learn the difference between dedicated and shared GPU memory, monitor VRAM overflow via Task Manager, and use quantization and context length control to avoid OOM.

GPT-OSS Open-Source Model Local Deployment Tutorial: Real-World Performance and VRAM Usage Analysis

Tutorials

May 15, 2026·3 min

GPT-OSS Open-Source Model Local Deployment Tutorial: Real-World Performance and VRAM Usage Analysis

Complete guide to deploying OpenAI's open-source GPT-OSS model locally with Ollama. Real-world testing of the 20B version on RTX 4090 covering Chinese comprehension, logical reasoning, and VRAM usage analysis under MoE architecture.