[KongchangAI]
Concept

cross-position supervision transfer

In autoregressive models, the supervision signal at one response position can improve the model's prediction quality at other positions, especially neighboring or deeper ones, owing to shared underlying representations and parameters.

Timeline (last 90 days)

Nov 17

In autoregressive models, gradient updates at different response positions share the underlying representation and parameters, so supervision of a certain deep token can transfer to other positions; this phenomenon is called cross-position supervision transfer.

Unverified50%

All Facts (1)

Source Articles