TensorRT
A high-performance deep learning inference optimizer and runtime released by NVIDIA since 2015. It supports operator fusion, precision calibration (FP32/FP16/INT8/FP8), automatic kernel tuning, and dynamic shapes, and is the de facto standard toolchain for GPU inference deployment.
Core Facts
TensorRT is a high-performance inference optimization engine provided by NVIDIA that compiles PyTorch/ONNX models into GPU-optimized execution engines, typically improving inference speed by 2–5×.
NVIDIA's CUDA ecosystem is the de facto standard in the AI inference industry.
CUDA是NVIDIA于2006年推出的并行计算平台和编程模型
Timeline (last 90 days)
NVIDIA AI Enterprise is an end-to-end software suite for enterprise data centers, including certified and optimized inference and training frameworks (such as TensorRT and Triton Inference Server), pretrained models and toolchains, running on NVIDIA GPUs.
Edge deployment involves model compression (quantization, pruning, knowledge distillation), specialized acceleration hardware (such as NVIDIA Jetson, Google Coral, neural processing units NPU), and the selection and optimization of inference frameworks (TensorRT, ONNX Runtime, TFLite).
NVIDIA currently holds a near-monopoly on the AI training and inference chip market, with its CUDA ecosystem forming an extremely high software moat.
The project uses TensorRT FP16 quantization and an input resolution of 512×512, with a current frame rate of 20–30 FPS.
JetPack SDK is NVIDIA's complete software stack for the Jetson platform, including the CUDA toolchain, cuDNN, TensorRT, computer vision libraries, and more.
OpenDLSS NR provides a browser version that runs using WebGPU and does not depend on TensorRT.
Extracting weights from NVIDIA's closed-source runtime may involve technical methods such as driver-level hooks and memory dumps, which may implicate reverse engineering-related legal provisions in certain jurisdictions.
The agent autonomously completed the full pipeline of installing DeepStream, obtaining the YOLO11 model, exporting it in ONNX format, and having TensorRT build a Jetson-optimized engine, with the engine build alone taking approximately 10 minutes.
ONNX is a framework-agnostic neural network model exchange format; TensorRT takes an ONNX model and performs optimizations such as layer fusion and operator selection based on the target GPU, generating a device-specific serialized execution engine.
TensorRT生成的序列化引擎是平台绑定的，在特定GPU型号和TensorRT版本下编译生成，不可跨平台直接复用
37 more timeline events
All Facts (20)
TensorRT is a high-performance inference optimization engine provided by NVIDIA that compiles PyTorch/ONNX models into GPU-optimized execution engines, typically improving inference speed by 2–5×.
90%VerifiedNVIDIA's CUDA ecosystem is the de facto standard in the AI inference industry.
85%VerifiedCUDA是NVIDIA于2006年推出的并行计算平台和编程模型
80%VerifiedTensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
80%VerifiedTensorRT's operator fusion merges multiple consecutive small operators in a computation graph into a single large operator, reducing the number of GPU kernel launches and memory reads/writes of intermediate results; a typical fusion operation can combine convolution, batch normalization, and activation function into a single GPU kernel call.
80%VerifiedTriton Inference Server is an open-source model serving framework from NVIDIA that supports dynamic batching and concurrent model execution, and is compatible with inference backends such as TensorRT, PyTorch, TensorFlow, and ONNX Runtime.
80%VerifiedTensorRT-LLM is an open-source framework developed by NVIDIA specifically for optimizing large language model inference.
75%VerifiedModel compression techniques such as INT8 quantization and channel pruning can increase inference speed by 2–4 times with a precision loss of less than 1% mAP
70%VerifiedINT8 quantization maps 32-bit floating-point weights to 8-bit integers, compressing model size by 75% with almost no accuracy loss
65%VerifiedMixed precision training can theoretically reduce memory usage by nearly half and achieve 2–8× computational speedup on GPUs that support Tensor Cores, such as T4, V100, and A100.
65%VerifiedTensorRT is an inference engine developed by NVIDIA, and ONNX Runtime is an inference engine developed by Microsoft.
65%UnverifiedTensorRT optimizes inference performance through layer fusion, precision calibration, and automatic kernel tuning
90%UnverifiedTensorRT能针对CUDA和GPU优化张量运算，进一步提升性能
90%UnverifiedTensorRT的内核自动调优机制会针对目标GPU的具体架构（如Ampere、Hopper等）对候选内核进行性能基准测试
90%UnverifiedTensorRT提供了熵校准（Entropy Calibration）和最小最大值校准（MinMax Calibration）等多种量化校准策略
90%UnverifiedTensorRT only supports the NVIDIA ecosystem
85%UnverifiedTensorRT内置了数百种融合模式的识别规则，能够自动发现并应用层融合优化
80%UnverifiedTensorRT优化后的模型通常比通用框架推理快2-6倍
75%UnverifiedTensorRT的层融合优化中，以Conv-BN-ReLU组合为例，融合后内存访问量可减少60%以上
75%UnverifiedTensorRT is NVIDIA's official model inference optimization library, which compresses and accelerates models through techniques such as operator fusion and precision quantization (FP16/INT8).
70%