Latent Factor Decomposition
A structured model compression approach that assumes weight matrices contain low-rank redundant structure. The method learns shared latent factors and combination rules to reconstruct original weights, enabling an average per-parameter storage cost of less than one bit, thereby achieving sub-1-bit model compression.
Timeline (last 90 days)
Sub-1-bit compression means that the average amount of information occupied per parameter is compressed to less than 1 bit.
The idea behind latent factor decomposition is to assume that weight matrices have redundancy and low-rank structure, and to reconstruct the original weights using a set of smaller latent factors along with indices or combination methods.
Lottery ticket subnetworks often exhibit low-rank or sparse characteristics, which closely align with the low-rank structure exploited by latent factor decomposition.
Traditional INT8 quantization can perform inference directly using hardware-native integer arithmetic units, whereas factor decomposition schemes require first reconstructing approximate weights from indices before executing multiplication.
All Facts (4)
Sub-1-bit compression means that the average amount of information occupied per parameter is compressed to less than 1 bit.
50%UnverifiedThe idea behind latent factor decomposition is to assume that weight matrices have redundancy and low-rank structure, and to reconstruct the original weights using a set of smaller latent factors along with indices or combination methods.
50%UnverifiedLottery ticket subnetworks often exhibit low-rank or sparse characteristics, which closely align with the low-rank structure exploited by latent factor decomposition.
50%UnverifiedTraditional INT8 quantization can perform inference directly using hardware-native integer arithmetic units, whereas factor decomposition schemes require first reconstructing approximate weights from indices before executing multiplication.
50%