Conceptentropy-regularized loss
Entropy Regularization Loss
Entropy regularization loss is a regularization technique that incorporates a routing-distribution entropy penalty term into the loss function. By penalizing low-entropy (concentrated) distributions, it prevents routing collapse and encourages dynamic routing mechanisms to maintain diverse allocation decisions.
Timeline (last 90 days)
Oct 8
The paper introduces an entropy-regularized loss to prevent the routing distribution from becoming overly concentrated, encouraging a more diverse routing distribution.
Unverified50%