#TensorRT-LLM
39 related articles
·3 min
Opus 5 Hands-On: Powerful but Slow — Inference Speed Becomes the New AI Battleground
An AI practitioner tests Opus 5: powerful capabilities but painfully slow inference. Deep analysis of how LLM competition is shifting from intelligence to inference speed.
Read more →
·3 min
OpenAI Removes the 5-Hour Limit: Intensifying AI Subscription Competition Makes Users the Biggest Winners
OpenAI removes the controversial 5-hour usage limit and raises subscription quotas. As competition with Anthropic, Google, and others intensifies, users are winning big in the AI arms race.
Read more →
·2 min
Production LLM Deployment Guide: Model Selection, GPU Configuration, and Inference Engine Breakdown
A comprehensive decision framework for production LLM deployment: model selection (open-source vs. API), GPU VRAM configuration, and inference engine comparison including vLLM and TensorRT-LLM.
Read more →
·4 min
From Enter to First Token: Decrypting the Full LLM Inference Pipeline (Autoregression, KV Cache & Decoding Strategies)
From pressing Enter to the first character appearing, what happens inside an LLM? This article breaks down autoregressive generation, KV cache acceleration, and decoding strategies like temperature, Top-k, and Top-p.
Read more →
·2 min
Meta Launches Muse Spark 1.1: AI Coding Model Now Open for API Access
Meta launches Muse Spark 1.1 and the Meta Model API, entering the AI coding market. The model plugs into existing dev tools, competing with GitHub Copilot and OpenAI.
Read more →
·5 min
The Five Unsolved Challenges of Production MLOps: A Deep Dive into GPU Scheduling, Fault-Tolerant Training, and Model Observability
A deep dive into the five genuinely tough challenges of production MLOps: fault-tolerant training on Spot instances, cross-team GPU scheduling, data reproducibility, model observability, and inference cost optimization.
Read more →
·4 min
Unsloth Releases Qwen3.6 NVFP4 Quantization: 2.5x Faster with Zero Accuracy Loss
Unsloth releases NVFP4 quantization for Qwen3.6 using W4A4 true 4-bit Tensor Core computation, delivering up to 2.5x inference speedup over NVIDIA's official implementation with accuracy matching or exceeding BF16 on benchmarks like MMLU-Pro.
Read more →
·3 min
PUBG Ally Deep Dive: How NVIDIA ACE Creates AI Teammates That Truly Collaborate
KRAFTON partners with NVIDIA ACE to build PUBG Ally, a next-gen AI teammate with voice recognition, LLM reasoning, and real-time tactical decisions—pioneering the evolution from NPC to CPC.
Read more →
·6 min
A Deep Dive into the vLLM Inference Framework: Core Principles of Throughput Optimization and Interview Guide
An in-depth analysis of the vLLM inference framework's core principles: from the meaning of throughput (tokens/s), to the bottlenecks of autoregressive generation, to KV Cache, PagedAttention, and continuous batching.
Read more →
·6 min
GPT-5.6 Sol Limited Preview: The Safety Boundaries Behind AI's Capability Expansion
OpenAI's flagship GPT-5.6 advances on three fronts—Sol, Kara, Luna tiered rollout; ByteDance CGN 5.0 Pro and Meta Muse push image generation toward controllable workflows; AI coding agents expose new supply chain risks.
Read more →
·6 min
GPT Live Full-Duplex, Grok 4.5 Coding, Seedream 5.0 Images: AI Accelerates Across Three Tracks Simultaneously
OpenAI's GPT Live full-duplex voice model, Grok 4.5 coding model with Cursor, and ByteDance's Seedream 5.0 Pro image generation launched together. A deep dive into three AI releases.
Read more →
·3 min
ComfyUI-INT4-Fast: Running Flux Models Smoothly on 6GB GPUs
ComfyUI-INT4-Fast brings W4A4 quantized inference to ComfyUI. RTX 3060 (6GB VRAM) generates 1024×1024 images in 17s. Per-layer mixed-precision routing balances speed and quality for Flux models.
Read more →
·5 min
Ollama Raises $65M Series B: 85% of Fortune 500 Have Already Deployed Local LLMs
Local LLM tool Ollama closes a $65M Series B, bringing total funding to $88M. With 9M developers and 85% of Fortune 500 having deployed internally, this deep dive explores why enterprises embrace local LLMs: compliance, Agent cost savings, and open-source ecosystem.
Read more →
·5 min
GPT-5.6 Trio Released: A Deep Dive into Sol/Terra/Luna and the Breakthrough in Design Judgment
OpenAI officially releases the GPT-5.6 series with three models: flagship Sol, balanced Terra, and economy Luna. A deep dive into its core breakthroughs—a step change in design judgment and enhanced computer-use capabilities.
Read more →
·4 min
Kun: An Open-Source AI Coding Agent Deeply Optimized for DeepSeek
Kun is an open-source AI coding agent optimized for DeepSeek and domestic users, with nearly 5,000 GitHub stars. Features include requirements drafting, inline diffs, cost visualization, and mobile monitoring. Real-world cache hit rates reached 97%, keeping costs extremely low.
Read more →
·5 min
DeepSeek Boosts Inference Speed by 85%: How Speculative Decoding Is Reshaping the AI Speed Race
DeepSeek and Peking University's DS Spark paper boosts AI inference speed by up to 85% via confidence scheduling and semi-autoregressive speculative decoding — no model or GPU changes.
Read more →
·5 min
TensorRT Multi-Device Inference: Engineering Practices for Breaking Through Single-GPU Memory Bottlenecks
NVIDIA TensorRT now supports multi-device inference via pipeline and tensor parallelism, distributing large models across multiple GPUs to break through single-card memory limits.
Read more →
·7 min
The Agent Engineer Transition Roadmap: A Deep Dive into Three Core Skill Systems
Want to become an Agent engineer? This article systematically covers three core skill tracks—LLM fundamentals, LangChain architecture development, and enterprise deployment—to help you avoid detours.
Read more →
·6 min
Xiaomi XiaoAI 10.1-inch Smart Control Panel: Is the AI LLM Integration Worth It?
Xiaomi XiaoAI 10.1-inch Smart Control Panel features AI LLM Q&A, WeChat calling, and whole-home Mi IoT control. Priced at 839 yuan, ~679 yuan after national subsidy. An in-depth review of AI capabilities, screen experience, and smart home integration.
Read more →
·5 min
DeepSeek DSpec Hands-On: How Speculative Decoding Accelerates LLM Inference by 6x
Hands-on report on DeepSeek's open-source inference acceleration toolkit DSpec: draft model + smart scheduling delivers lossless speedup, hitting acceptance length 6 on GSM8K and reproducing official data.
Read more →