SkyRL Delivers Day-0 RL Support for Beam, Boosting LoRA Decode Throughput by 62%

Title

SkyRL Delivers Day-0 RL Support for Beam, Boosting LoRA Decode Throughput by 62%

Meta Description

SkyRL announces Day-0 reinforcement learning support for Reflection's Beam model, covering LoRA, BF16 full fine-tuning, and MXFP8 RL, combined with vLLM rollout and Megatron-Bridge, achieving a 62% increase in LoRA decode throughput to 447 tok/s/GPU.

Article Content

The SkyRL team has announced Day-0 reinforcement learning support for Reflection's Beam model, meaning Beam can be integrated into a complete RL training pipeline on the day of its release. This advancement comes from a collaboration between SkyRL and Reflection, with the core breakthrough being the unification of LoRA, BF16 full fine-tuning, and MXFP8 reinforcement learning within the vLLM inference and Megatron-Bridge training frameworks.

Tech Stack Integration: Synergy Between vLLM and Megatron-Bridge

SkyRL's support covers three training modes: LoRA fine-tuning, BF16 full fine-tuning, and MXFP8 low-precision reinforcement learning. On the inference side, vLLM is used for rollout generation, while the training side relies on Reflection's Megatron-Bridge. This combination allows the Beam model to complete the closed loop from sampling to updating within a single infrastructure.

You might not have noticed that R3 (SkyRL's distributed training solution) can already work in conjunction with Pipeline Parallelism. During weight synchronization, FP32 router weights are fully preserved, avoiding numerical instability caused by low-precision conversion.

Differences Between LoRA, Full Fine-Tuning, and Low-Precision RL

LoRA (Low-Rank Adaptation) trains by attaching low-rank matrices alongside weights, freezing the original model parameters. Its memory footprint and communication overhead are far smaller than full fine-tuning, making it suitable for rapid iteration and multi-task switching. BF16 full fine-tuning updates all parameters, offering higher precision but with significant memory and bandwidth demands, typically used when pursuing final performance. MXFP8 is an 8-bit floating-point micro-scaling format where each small block shares an exponent scaling factor, compressing weights and activations to 8 bits while maintaining dynamic range. It can significantly improve compute and memory efficiency but is more sensitive to numerical stability, thus requiring safeguards such as FP32 router weights. Unifying all three into the same toolchain means teams can flexibly choose between cost, speed, and precision based on task requirements without switching training frameworks.

Source of the 62% Decode Throughput Improvement

SkyRL's official report provides a key data point: by properly configuring expert adapters in vLLM, LoRA decode throughput increased from 276 tok/s/GPU to 447 tok/s/GPU, achieving approximately 62% growth under a configuration of 32 concurrent requests and PP2×TP8.

It is important to clarify that this is a decode-phase result, not an end-to-end training speedup ratio. The improvement in decode throughput directly affects the efficiency of RL rollout, as a significant amount of time in reinforcement learning training is spent generating samples. For models like Beam with a Mixture-of-Experts (MoE) architecture, the configuration of expert adapters significantly impacts inference efficiency. SkyRL's tuning experience provides a reusable reference for similar scenarios.

Expert Adapters and MoE Inference

Expert adapters are LoRA adapters attached separately to each expert in an MoE model. Unlike dense models that have only one set of adapters, each expert in an MoE has independent weights. During inference, tokens are routed to a small number of experts. If adapters are organized improperly, it can cause frequent kernel launches, fragmented memory access, and reduced batching efficiency. vLLM improves decode throughput without changing the model structure by batching adapters across experts and scheduling them fused with expert computation to reduce overhead. This also explains why the 62% improvement is highly configuration-dependent: the number of experts, routing distribution, and concurrency all affect adapter scheduling efficiency.

Significance for RL Training Workflows

Day-0 support means a new model can enter the RL post-training pipeline on the day of release without waiting for framework adaptation. SkyRL unifies LoRA, full fine-tuning, and MXFP8 into a single toolchain, reducing the cost for researchers and engineering teams to switch training modes.

From an engineering perspective, the combination of vLLM rollout and Megatron-Bridge training balances inference throughput and training stability. The preservation of FP32 router weights indicates that the team did not sacrifice the numerical precision of MoE routing while pursuing low-precision training efficiency.

What Day-0 Support Means

Day-0 support means that on the day a new model is released, the training framework has already completed adaptation and can be directly used for RL post-training. In traditional workflows, after a new model is released, framework developers need weeks to adapt weight formats, parallel sharding, and inference kernels, during which researchers can only wait. Achieving Day-0 requires the framework to know the model structure in advance and reuse the generic interfaces of vLLM and Megatron, so that a new model can be integrated by simply providing a configuration. This is especially important for RL post-training because RL relies on extensive online rollouts, and the integration cost between inference and training frameworks is often higher than for supervised fine-tuning.

Outlook

This update from SkyRL reflects the trend of RL training frameworks evolving toward "support upon model release." As MoE architectures and low-precision training become mainstream, how to efficiently synchronize weights between vLLM inference and Megatron training, and how to perform fine-grained tuning for expert adapters, will become key competitive points in RL infrastructure. The decode throughput data and configuration details published by SkyRL provide the community with a valuable practical benchmark.

One-line Summary

SkyRL provides Day-0 reinforcement learning support for the Beam model, unifying three training modes and achieving a 62% increase in decode throughput.

Paragraph Summary

The SkyRL team announced a collaboration with Reflection to deliver Day-0 reinforcement learning support for the Beam model, enabling integration into a complete RL training pipeline on the day of release. This support unifies LoRA fine-tuning, BF16 full fine-tuning, and MXFP8 low-precision reinforcement learning within the vLLM inference and Megatron-Bridge training frameworks, forming a closed loop from rollout sampling to parameter updates. The R3 distributed solution can work with pipeline parallelism and preserves FP32 router weights during weight synchronization to maintain numerical stability of MoE routing. Official data shows that by properly configuring vLLM expert adapters, LoRA decode throughput increased from 276 to 447 tok/s/GPU, an increase of about 62%, but this is a decode-phase result rather than an end-to-end training speedup. This advancement lowers the cost of switching training modes and provides a reusable benchmark for MoE and low-precision RL infrastructure.