#TensorRT inference

98 related articles

Jul 10, 2026·5 min

TensorRT Multi-Device Inference: Engineering Practices for Breaking Through Single-GPU Memory Bottlenecks

NVIDIA TensorRT now supports multi-device inference via pipeline and tensor parallelism, distributing large models across multiple GPUs to break through single-card memory limits.

Jul 30, 2026·3 min

Is LLM Inference Really That Expensive? Breaking Down the Cost Reduction Logic of GPU Compute and Inference Frameworks

Analyzing real LLM inference costs: from B200 GPU compute gains, vLLM framework optimization to MTP multi-token prediction, explaining why serving costs are widely overestimated.

Jul 30, 2026·2 min

Behind the Open-Source Model Frenzy: Who Will Provide Cheap Inference Services?

Open-source LLM weights don't equal low-cost access for developers. This article analyzes the inference service gap in open-source AI and how providers like Together AI and Groq are addressing it.

Jul 30, 2026·2 min

Behind the Open-Source Model Frenzy: Who Will Provide Cheap Inference Services?

Open-source LLM weights don't mean developers can use them cheaply. This article examines the inference service gap in open-source AI and how providers like Together AI and Groq are addressing it.

Jul 29, 2026·3 min

Production-Grade LLM Inference Deployment: A Practical K8s Handbook

Complete guide to deploying production-grade LLM inference on Kubernetes, covering GPU scheduling, vLLM engine selection, autoscaling, observability, and cost optimization.

Jul 29, 2026·3 min

Moonshot AI Open-Sources FlashKDA: A Deep Dive into High-Performance CUDA Kernels for KDA Attention

Moonshot AI open-sources FlashKDA, providing high-performance CUDA kernels for Kimi Delta Attention. Explore its technical principles, performance gains, and value for long-context training and inference.

Jul 29, 2026·3 min

Moonshot AI Open-Sources FlashKDA: A Deep Dive into High-Performance CUDA Kernels for KDA Attention

Moonshot AI open-sources FlashKDA, providing high-performance CUDA kernels for Kimi Delta Attention. Learn about its technical principles, performance gains, and value for long-context training and inference acceleration.

Jul 28, 2026·3 min

ncnn Vulkan Cross-Platform Edge Inference in Practice: Solving the GPU Fragmentation Deployment Challenge

Facing GPU fragmentation on edge devices, the PostSlate team used ncnn's Vulkan backend for cross-platform ML inference, achieving 10× speedup on RTX 4070 with half the model size and zero runtime installation.

Jul 28, 2026·4 min

Cursor Start India Plan at ₹649/Month: Decoding the Regional Pricing Strategy for AI Coding Tools

Cursor launches its Start plan for India at ₹649/month (~$7.70), featuring Grok 4.5 and Composer Agent. A deep dive into the PPP pricing strategy and its market implications.

Jul 28, 2026·3 min

AMD CDNA5 Architecture Deep Dive: Technical Evolution and the AI Computing Competition Landscape

Deep analysis of AMD's CDNA5 architecture covering Chiplet packaging upgrades, HBM memory evolution, and low-precision compute optimization, examining how AMD challenges NVIDIA's AI chip dominance.

Jul 27, 2026·3 min

Jensen Huang's First Tweet Backs AI Open Source — But It's Really About NVIDIA's Ecosystem Anxiety

Jensen Huang's first tweet backs AI open source, but behind it lies NVIDIA's deep anxiety over CUDA ecosystem displacement. We analyze why open-source models matter and what's really at stake.

Jul 27, 2026·3 min

Jensen Huang's First X Post Backs Open AI: Why the Computing Power King Sides with Open Source

NVIDIA CEO Jensen Huang's first X post champions open AI access. We analyze the business logic, policy dynamics, and the open vs. closed AI debate shaping the industry.

Jul 27, 2026·2 min

Jensen Huang Champions Open-Source AI: Distillation Is Not Plagiarism, DeepSeek and Kimi Are Excellent

NVIDIA CEO Jensen Huang defends open-source AI, calls distillation legitimate learning, praises DeepSeek and Kimi, and co-signs open letter with 20+ companies while OpenAI and Google stay silent.

Jul 27, 2026·2 min

RX 9060 XT vs RTX 5060 Ti: How to Choose a GPU for Local AI

RX 9060 XT vs RTX 5060 Ti both offer 16GB VRAM — which is better for local AI inference? A full comparison of CUDA ecosystem, ROCm compatibility, LLM performance, and real-world usability.

Jul 27, 2026·2 min

RX 9060 XT vs RTX 5060 Ti: Which GPU to Pick for Local AI?

RX 9060 XT vs RTX 5060 Ti — both 16GB VRAM, but which is better for local AI? We compare CUDA ecosystem, ROCm compatibility, LLM inference, and real-world usability.

Jul 27, 2026·3 min

Building a YOLO26 Inference Engine from Scratch in ARM64 Assembly: A Deep Dive into Low-Level Optimization

Deep dive into building a YOLO26n object detection inference engine from scratch using ARM64 assembly and C, covering NEON SIMD, Winograd convolution, GEMM micro-kernels, and cache tiling optimizations.

Jul 27, 2026·3 min

Opus 5 Hands-On: Powerful but Slow — Inference Speed Becomes the New AI Battleground

An AI practitioner tests Opus 5: powerful capabilities but painfully slow inference. Deep analysis of how LLM competition is shifting from intelligence to inference speed.

Jul 27, 2026·7 min

DIY Companion Robot: Building Your AI Desktop Buddy with Jetson Orin

A maker builds a DIY companion robot with NVIDIA Jetson Orin and 4S LiPo battery. Explore the full development journey from first power-up to AI interaction, including edge computing, power design, and companion robot trends.

Intelligent Model Routing: The Core Te…

Jul 24, 2026·3 min

Intelligent Model Routing: The Core Technology and Implementation Guide for AI Cost Reduction and Efficiency

Intelligent Model Routing is becoming key AI infrastructure. This article explores its principles, solution types, technical challenges, and implementation considerations to help developers balance cost, latency, and quality.

Jul 24, 2026·6 min

AI Application Engineer Interview Guide: Core Topics in Model Quantization and Inference Deployment

A systematic review of must-know topics for AI Application Engineer interviews: PTQ/QAT quantization, operator fusion, inference pipelines, latency/throughput analysis, and edge deployment of detection/segmentation/BEV models.