Triton
An open-source domain-specific language (DSL) developed by OpenAI for writing GPU operators, designed to simplify the authoring of high-performance GPU compute kernels. It is widely used in deep learning inference frameworks.
Core Facts
Triton is a GPU programming language developed by OpenAI.
Unsloth is an open-source tooling team focused on large model quantization and fine-tuning acceleration, providing GGUF quantized versions for mainstream open-source models.
Triton is an open-source project by OpenAI that enables developers to write high-performance GPU kernels using Python-like syntax, lowering the barrier to CUDA programming.
Timeline (last 90 days)
MoE's dynamic routing causes the number of tokens received by each expert and each batch to change constantly, making shape combinations highly prone to divergence, which leads to a low Triton compilation cache hit rate and thus causes long-tail latency and first-token jitter.
Common approaches to mitigating repeated Triton JIT compilation include precompiling and caching common shapes, bucketing and merging shapes, fixing the maximum shape together with masking, and warming up all possible kernel variants at deployment time.
By reducing the frequency of Triton JIT recompilation, the AWS team contributed significantly to the improvement in time to first token (TTFT), but the original text did not disclose specific implementation details.
The AWS engineering team carried out kernel-level optimizations on Amazon SageMaker AI for a fused MoE model, including Swap-AB to improve WGMMA utilization, Triton backend routing to implement FP8 quantization, and eliminating repeated Triton JIT compilation.
Triton is a Python-like kernel DSL for GPUs that uses just-in-time compilation (JIT), generating and compiling PTX/CUBIN on first invocation based on specialization keys such as tensor shapes and data types.
The AWS team used Triton's backend routing mechanism to let different computation paths go to the backend implementations suited to each, thereby maintaining numerical stability under FP8 quantization.
When generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
Triton is a high-level programming language for GPUs in which developers describe computation in units of tiles, and the compiler is responsible for mapping to threads, shared memory, and registers.
Compared with CUDA, Triton leaves low-level details such as thread scheduling and coalesced memory access to the compiler, while retaining the ability to tune key parameters such as tile shape, num_warps, and num_stages.
The core idea of FBTriton is to re-express TBE's forward and backward computation using Triton, a high-level GPU programming language.
40 more timeline events
All Facts (20)
Triton is a GPU programming language developed by OpenAI.
90%VerifiedUnsloth is an open-source tooling team focused on large model quantization and fine-tuning acceleration, providing GGUF quantized versions for mainstream open-source models.
85%VerifiedTriton is an open-source project by OpenAI that enables developers to write high-performance GPU kernels using Python-like syntax, lowering the barrier to CUDA programming.
75%VerifiedNVIDIA's software stack from the bottom layer to the application layer includes CUDA, cuDNN, cuBLAS, TensorRT, TensorRT-LLM, Triton Inference Server, and NIM.
75%UnverifiedUnsloth使用手写Triton内核替代PyTorch默认实现来优化训练速度
90%UnverifiedUnsloth使用Triton语言编写了自定义GPU算子，针对Transformer架构中的注意力计算、矩阵乘法等核心操作进行了深度优化
85%UnverifiedUnsloth团队使用Triton（OpenAI开源的GPU编程语言）手动重写了Transformer模型中关键算子的反向传播内核
85%UnverifiedUnsloth's performance improvements come from hand-written GPU kernels using OpenAI's Triton language.
70%UnverifiedBy reducing the frequency of Triton JIT recompilation, the AWS team contributed significantly to the improvement in time to first token (TTFT), but the original text did not disclose specific implementation details.
50%UnverifiedMoE's dynamic routing causes the number of tokens received by each expert and each batch to change constantly, making shape combinations highly prone to divergence, which leads to a low Triton compilation cache hit rate and thus causes long-tail latency and first-token jitter.
50%UnverifiedCommon approaches to mitigating repeated Triton JIT compilation include precompiling and caching common shapes, bucketing and merging shapes, fixing the maximum shape together with masking, and warming up all possible kernel variants at deployment time.
50%UnverifiedThe AWS team used Triton's backend routing mechanism to let different computation paths go to the backend implementations suited to each, thereby maintaining numerical stability under FP8 quantization.
50%UnverifiedThe AWS engineering team carried out kernel-level optimizations on Amazon SageMaker AI for a fused MoE model, including Swap-AB to improve WGMMA utilization, Triton backend routing to implement FP8 quantization, and eliminating repeated Triton JIT compilation.
50%UnverifiedTriton is a Python-like kernel DSL for GPUs that uses just-in-time compilation (JIT), generating and compiling PTX/CUBIN on first invocation based on specialization keys such as tensor shapes and data types.
50%UnverifiedWhen generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
50%UnverifiedThe core idea of FBTriton is to re-express TBE's forward and backward computation using Triton, a high-level GPU programming language.
50%UnverifiedTriton is a high-level programming language for GPUs in which developers describe computation in units of tiles, and the compiler is responsible for mapping to threads, shared memory, and registers.
50%UnverifiedCompared with CUDA, Triton leaves low-level details such as thread scheduling and coalesced memory access to the compiler, while retaining the ability to tune key parameters such as tile shape, num_warps, and num_stages.
50%UnverifiedAcross microbenchmarks of five shapes, v3 (split-K + autotuned tile) is not faster than v2 (fused kernel) in any case, and v2 is 1.0 to 1.4 times faster than v3.
50%UnverifiedOn an RTX 3060, the developer's hand-written Triton fused W4A16 GEMV kernel targeting batch-1 decoding runs at about 25.5 tok/s, matching the roughly 25.7 tok/s of the bitsandbytes NF4 quantization scheme.
50%