[KongchangAI]
Concept

Mean-SVD

A method for extracting a language subspace from model activations. It performs singular value decomposition (SVD) on the mean activation vectors of different languages and takes the principal singular vectors as language directions.

Timeline (last 90 days)

Nov 4

The study used two methods for constructing language subspaces: the existing Mean-SVD construction and DiffPCA, proposed by the authors.

Unverified50%
Nov 4

Both the Mean-SVD and DiffPCA subspaces captured a considerable degree of translation-related variation and exhibited similar patterns across different model depths.

Unverified50%
Nov 4

After projecting out the Mean-SVD or DiffPCA subspaces, linear probes could still almost perfectly identify the language, indicating that neither method exhaustively captures language information.

Unverified50%

All Facts (3)

Source Articles