#delayed untying

nanoGPT Speedrun Techniques: How Delayed Untying Solves the Sparse Gradient Problem in Embedding Layers

Deep dive into the Delayed Untying technique in nanoGPT speedruns: why tying embed and lm_head weights early then untying later solves both sparse gradients and limited expressiveness.