#TensorRT-LLM
43 related articles
·5 min
TensorRT Multi-Device Inference: Engineering Practices for Breaking Through Single-GPU Memory Bottlenecks
NVIDIA TensorRT now supports multi-device inference via pipeline and tensor parallelism, distributing large models across multiple GPUs to break through single-card memory limits.
Read more →
·7 min
The Agent Engineer Transition Roadmap: A Deep Dive into Three Core Skill Systems
Want to become an Agent engineer? This article systematically covers three core skill tracks—LLM fundamentals, LangChain architecture development, and enterprise deployment—to help you avoid detours.
Read more →
·6 min
Xiaomi XiaoAI 10.1-inch Smart Control Panel: Is the AI LLM Integration Worth It?
Xiaomi XiaoAI 10.1-inch Smart Control Panel features AI LLM Q&A, WeChat calling, and whole-home Mi IoT control. Priced at 839 yuan, ~679 yuan after national subsidy. An in-depth review of AI capabilities, screen experience, and smart home integration.
Read more →
·5 min
DeepSeek DSpec Hands-On: How Speculative Decoding Accelerates LLM Inference by 6x
Hands-on report on DeepSeek's open-source inference acceleration toolkit DSpec: draft model + smart scheduling delivers lossless speedup, hitting acceptance length 6 on GSM8K and reproducing official data.
Read more →
·4 min
NVFP4 Quantization in Practice: A Guide to Compressing Large Models with NVIDIA Model Optimizer
Deep dive into NVFP4 quantization: using NVIDIA Model Optimizer to compress Nemotron 3 Ultra to FP4 checkpoints, reducing memory by 75% and boosting inference throughput on Blackwell GPUs.
Read more →
·5 min
NVIDIA AI-Q Blueprint × OCI: A Practical Guide to Production Deployment of Enterprise AI Agents
Deep dive into NVIDIA AI-Q Blueprint production deployment on Oracle Cloud Infrastructure, covering NIM microservices, RAG architecture, multi-agent orchestration, and OCI GPU selection for enterprise AI agents.
Read more →
·5 min
NVIDIA AI-Q Blueprint × OCI: A Practical Guide to Production Deployment of Enterprise AI Agents
Deep dive into NVIDIA AI-Q Blueprint production deployment on Oracle Cloud Infrastructure, covering NIM microservices, RAG architecture, multi-agent orchestration, and OCI GPU selection.
Read more →
From a Single Prompt to an AI Product:…
·3 min
From a Single Prompt to an AI Product: Starting Is Easy, Shipping Is Hard
AI startups begin with a prompt, but going from idea to product means overcoming major technical, product, and business challenges. A low barrier to entry doesn't mean a low barrier to success.
Read more →
·3 min
NVIDIA ACE SDK: On-Device AI Inference for Intelligent Game NPC Companions
Deep dive into NVIDIA ACE Game Agent SDK's integration with Unreal Engine 5, exploring how on-device AI inference enables low-latency, privacy-safe intelligent NPC dialogue and behavior.
Read more →
·3 min
748-Episode AI Large Language Model Development Tutorial: Seven Core Modules from Zero to Project Deployment
Analysis of a 748-episode, 198-hour AI LLM development tutorial covering API integration, prompt engineering, RAG, AI Agents, fine-tuning, multimodal development, and deployment.
Read more →
·2 min
Baseten Raises $1.5 Billion: Why AI Inference Infrastructure Has Become a Capital Darling
AI inference startup Baseten is raising $1.5B at a $130B valuation. We analyze why inference infrastructure is booming, the competitive landscape, and what this mega-round signals.
Read more →
·4 min
Local Deployment of Claude Code: Principles and Practical Guide for Three Approaches
A detailed guide to locally deploying Claude Code with three approaches (LM Studio, Ollama, vLLM), covering architecture, protocol translation, hardware selection, and model recommendations.
Read more →
·2 min
A Complete Guide to LLM Infrastructure: Core Challenges from GPU Clusters to Inference Optimization
A deep dive into core challenges and key technologies for LLM infrastructure, covering GPU cluster management, inference optimization, distributed training, cost control, and observability.
Read more →
·4 min
AI Engineer Job Search: The Capability Leap from Demo to Production System
AI job demand is surging but companies can't find qualified candidates. Learn the 3 core skills—advanced RAG, local model deployment, and full-stack monitoring—to leap from demo builder to production engineer.
Read more →
·4 min
The Five-Tier Pyramid of IT Careers in the AI Era: Your Position Determines Your Career Ceiling
AI is reshaping IT careers into a five-tier pyramid from tool usage to self-developed models. Learn where you fit and how to maximize your career potential.
Read more →
·3 min
vLLM Deep Dive: How PagedAttention Enables High-Throughput LLM Inference
Deep dive into vLLM's core technologies for high-throughput LLM inference, including PagedAttention memory management, continuous batching, distributed deployment, and comparisons with TensorRT-LLM.
Read more →
Tutorials·1 min
AI Large Language Model Learning Roadmap: Six Stages from Zero to Engineer
A systematic LLM engineer learning roadmap covering Transformer basics, prompt engineering, RAG, Agent development, API integration, fine-tuning, deployment, and project practice across six stages.
Read more →
Tech Frontiers·1 min
Windsurf Integrates Claude Opus 4.7 Fast Mode with 2.5x Speed Boost
Windsurf integrates Claude Opus 4.7 fast mode with 2.5x speed boost while retaining full intelligence. Analysis of its impact on developer productivity and AI coding tool competition.
Read more →
Industry Insights·2 min
AMD MI355X Beats B200: Full-Stack Optimization Breakdown for 5% Lower TCO on DeepSeek-R1 Inference
AMD Instinct MI355X achieves 5% lower TCO than NVIDIA B200 on DeepSeek-R1 disaggregated inference via SGLang+MoRI full-stack optimization with 1.25x per-GPU throughput.
Read more →
Tech Frontiers·1 min
SGLang Hosts Agent Loops Office Hour, Focusing on Agentic Loop Architecture Optimization
SGLang team hosts an Agent Loops Office Hour exploring inference optimization for agentic loops, covering KV Cache reuse, low-latency multi-turn dialogue, and tool calling techniques.
Read more →