RC-DS
An SAE training method that removes the singular value gain of the readout transformation via singular value decomposition, retaining only the orthogonal or rotational component. The SAE is trained in normalized decision-space coordinates, preventing the training objective from being distorted by readout scaling.
Timeline (last 90 days)
In the setting with condition number 100, when the factors are generated before the readout scaling, RC-DS raises the average sparse factor recovery rate from 0.506 to 0.913, against a baseline of centered logit training under the same budget.
RC-DS implements an exact wrapper that reproduces RC-DS training to numerical precision, with the aim of isolating the effect of training geometry rather than introducing additional information.
In neural network policies, RC-DS improves the average logit reconstruction at the test budget.
On three fixed language model checkpoints, RC-DS achieves a higher gain-free reconstruction mean in all 18 task-sparsity settings.
The core operation of RC-DS is: first perform a singular value decomposition on the readout transformation, remove the singular value gains, keep only the orthogonal or rotational part that determines the directional structure, and then train the SAE in the normalized decision-space coordinates.
All Facts (5)
In the setting with condition number 100, when the factors are generated before the readout scaling, RC-DS raises the average sparse factor recovery rate from 0.506 to 0.913, against a baseline of centered logit training under the same budget.
50%UnverifiedRC-DS implements an exact wrapper that reproduces RC-DS training to numerical precision, with the aim of isolating the effect of training geometry rather than introducing additional information.
50%UnverifiedIn neural network policies, RC-DS improves the average logit reconstruction at the test budget.
50%UnverifiedOn three fixed language model checkpoints, RC-DS achieves a higher gain-free reconstruction mean in all 18 task-sparsity settings.
50%UnverifiedThe core operation of RC-DS is: first perform a singular value decomposition on the readout transformation, remove the singular value gains, keep only the orthogonal or rotational part that determines the directional structure, and then train the SAE in the normalized decision-space coordinates.
50%