Self-Training
A training paradigm in which a language model, over multiple generations of iteration, fine-tunes on its own outputs as training data; it is often used to study model collapse and distribution shift.
Timeline (last 90 days)
In self-training, the training temperature affects the sampling distribution during data generation: temperature T adjusts the model's output probabilities via p_i ∝ exp(z_i/T), with T>1 making the distribution flatter and T<1 making it sharper.
The study uses English GPT-2 for nine generations of self-training, fine-tuning each generation on the previous generation's outputs and varying the sampling temperature of the training data; all models are measured with the same fixed sampler to ensure fair comparison.
All Facts (2)
In self-training, the training temperature affects the sampling distribution during data generation: temperature T adjusts the model's output probabilities via p_i ∝ exp(z_i/T), with T>1 making the distribution flatter and T<1 making it sharper.
50%UnverifiedThe study uses English GPT-2 for nine generations of self-training, fine-tuning each generation on the previous generation's outputs and varying the sampling temperature of the training data; all models are measured with the same fixed sampler to ensure fair comparison.
50%