RFT
Reinforcement Fine-Tuning is a method for fine-tuning pre-trained language models using reinforcement learning signals. It updates model parameters through reward-weighted policy gradients and demonstrates certain advantages over supervised fine-tuning in preserving stylistic symmetry and mitigating catastrophic forgetting.
Timeline (last 90 days)
Rejection Fine-Tuning (RFT) is a post-training paradigm that requires no human preference annotations, originally popularized by Meta in the LLaMA 2 research.
Supervised fine-tuning (SFT) tends to cause more severe catastrophic forgetting than reinforcement fine-tuning (RFT).
SFT and RFT are parallel and consistent in semantic updates, but fundamentally different in style dynamics.
Starting from a policy with no intra-class style preference, RFT using exact policy gradients can maintain style symmetry without introducing style bias.
RFT is able to maintain zero semantic error.
RFT optimizes for correctness of results rather than token-by-token imitation of a specific style, thereby avoiding the side effect of style drift, which explains why RLHF/RFT is more robust in preserving general capabilities.
All Facts (6)
Rejection Fine-Tuning (RFT) is a post-training paradigm that requires no human preference annotations, originally popularized by Meta in the LLaMA 2 research.
50%UnverifiedSupervised fine-tuning (SFT) tends to cause more severe catastrophic forgetting than reinforcement fine-tuning (RFT).
50%UnverifiedSFT and RFT are parallel and consistent in semantic updates, but fundamentally different in style dynamics.
50%UnverifiedStarting from a policy with no intra-class style preference, RFT using exact policy gradients can maintain style symmetry without introducing style bias.
50%UnverifiedRFT is able to maintain zero semantic error.
50%UnverifiedRFT optimizes for correctness of results rather than token-by-token imitation of a specific style, thereby avoiding the side effect of style drift, which explains why RLHF/RFT is more robust in preserving general capabilities.
50%