Specialized Predictor
A predictor that achieves generalization by memorizing the topic structure in the training data, corresponding to a solution in which model parameters store the training distribution. It emerges only after the amount of training data exceeds the number of parameters of the data-generating process.
Timeline (last 90 days)
In this solvable topic model there exist two distinct solutions: the specialised predictor and the in-context predictor.
The specialised predictor stores the entire finite topic mixture distribution, which amounts to memorizing the topic structure that appears in training.
The core prediction of this theory is: the in-context predictor is available very early, already existing when the amount of training data is still small; whereas the specialised predictor only emerges after the number of training tokens exceeds the number of parameters of the data generation process.
All Facts (3)
In this solvable topic model there exist two distinct solutions: the specialised predictor and the in-context predictor.
50%UnverifiedThe specialised predictor stores the entire finite topic mixture distribution, which amounts to memorizing the topic structure that appears in training.
50%UnverifiedThe core prediction of this theory is: the in-context predictor is available very early, already existing when the amount of training data is still small; whereas the specialised predictor only emerges after the number of training tokens exceeds the number of parameters of the data generation process.
50%