DiffPCA
A method for extracting a language subspace from model activations. It first computes the activation differences of the same semantics across different languages, then applies PCA to the difference matrix to capture language-transformation directions.
Timeline (last 90 days)
The study used two methods for constructing language subspaces: the existing Mean-SVD construction and DiffPCA, proposed by the authors.
Both the Mean-SVD and DiffPCA subspaces captured a considerable degree of translation-related variation and exhibited similar patterns across different model depths.
After projecting out the Mean-SVD or DiffPCA subspaces, linear probes could still almost perfectly identify the language, indicating that neither method exhaustively captures language information.
All Facts (3)
The study used two methods for constructing language subspaces: the existing Mean-SVD construction and DiffPCA, proposed by the authors.
50%UnverifiedBoth the Mean-SVD and DiffPCA subspaces captured a considerable degree of translation-related variation and exhibited similar patterns across different model depths.
50%UnverifiedAfter projecting out the Mean-SVD or DiffPCA subspaces, linear probes could still almost perfectly identify the language, indicating that neither method exhaustively captures language information.
50%