Product-Key Memory

A neural network memory module based on key-value retrieval that organizes memory slots through the Cartesian product of two sets of key vectors; queries compute inner products with both key sets and weight the aggregation of value vectors, achieving O(N²) memory capacity at O(N) cost and often used to replace or supplement the FFN in Transformers.