ConceptDecoding by Contrasting Layers
DoLa
A decoding method that contrasts output distributions from different layers within a single model, amplifying factual knowledge carried by high-confidence layers while suppressing generalization errors that may arise in lower layers.
Timeline (last 90 days)
Oct 15
Decoding adjustments do not touch model weights or internal representations, but instead modify the token selection strategy during the generation stage; contrastive decoding runs a strong model and a weak model simultaneously and takes the difference in their log probabilities as the score for the next token; DoLa compares the output distributions of different layers within a single model.
Unverified50%