Unverified50% confidenceFactExact time
In early regression experiments, PreCap appeared far superior to SiLU and GELU at a fixed learning rate, but once each activation function was given its own tuned learning rate, this advantage disappeared entirely.
1
Sources
50%
Confidence
Long-term
Relevance
10/27/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedDuring joint fine-tuning, a lower learning rate helps stabilize policy updates and avoid destroying fine-grained behaviors that have already been learned.69% similarUnverifiedUsing a sparse reinforcement learning reward signal based solely on final outcomes, the model repaired the true-theorem recognition ability destroyed by SFT and raised the recognition rate to 0.66, surpassing the base model.67% similarUnverifiedOver-reliance on the fine-tuned part during fine-tuning may be precisely the root cause of rare-class capabilities being obscured; directing the model's computation moderately toward the pretrained components may be a more robust fine-tuning route.67% similarUnverifiedThe median improvement of 5.3 percentage points indicates that current models are mostly doing 'local patching' and have not yet formed cross-level knowledge accumulation, which is the key difference between continual learning and single-round self-correction.67% similarUnverifiedThe cosine annealing learning rate scheduling strategy was proposed by Loshchilov and Hutter in 2017 and smoothly decays the learning rate from its initial value to near zero following a cosine function.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1102907API
curl https://kongchang.com/api/v1/knowledge/claims/1102907MCP
get_claim(id=1102907)