No MLP or Attention Required: How Point-Attractor Dynamics Learn Word Embeddings

A Counterintuitive Experiment

In an era dominated by Transformers and attention mechanisms, one developer posed a nearly countercurrent question: Can a minimalist dynamical system learn useful word representations without any MLP, Transformer, or attention layer—not even a separate output matrix?

Dynamical systems are a framework in mathematics and physics for studying how states evolve over time. Originating in Poincaré's work on celestial mechanics, the field was later developed into modern nonlinear dynamics by Lyapunov and others. Introducing dynamical frameworks into machine learning is not entirely new: recurrent neural networks (RNNs) are essentially discrete-time dynamical systems, while Reservoir Computing and Echo State Networks explicitly borrow ideas from chaotic dynamics. Applying it to word representation learning means encoding semantic information via "state trajectories" rather than "static mappings"—traditional word embedding methods like word2vec and GloVe map words to fixed vectors, whereas the dynamical perspective focuses on the continuous evolution of a state vector across a word sequence. This approach stands in sharp contrast to the mainstream neural network paradigm.

This experiment, dubbed "Collapse / noun-collapse," offers an intriguing answer. On the SimLex-999 noun subset, it achieved a similarity correlation of Spearman ρ = 0.3616, covering 662/666 noun pairs.

SimLex-999 is a standard benchmark dataset in natural language processing for evaluating word similarity. Proposed by Hill et al. in 2015, it contains 999 English word pairs, each scored 0–10 for similarity by human annotators. Word similarity evaluation benchmarks evolved from WordSim-353 (Finkelstein et al., 2001) to MEN (Bruni et al., 2012). Early benchmarks conflated the concepts of similarity and relatedness—"doctor" and "nurse" are highly related but not entirely similar, while "coffee" and "cup" are related but not similar. SimLex-999 addressed this by giving explicit instructions requiring annotators to strictly distinguish these two concepts, contrasting the high similarity score of "cat" and "dog" with the low similarity score of "coffee" and "cup," making it a stricter benchmark for evaluating true semantic similarity. Spearman ρ (Spearman's rank correlation coefficient) measures the monotonic correlation between the model's predicted rankings and human judgment rankings, ranging from -1 to 1—the higher the value, the closer the model aligns with human semantic intuition.

The number itself isn't dazzling, but given the simplicity of the model's structure, its significance lies in proving a possibility: relying solely on raw co-occurrence pressure, ordered point-attractor dynamics can spontaneously learn a useful semantic geometry.

reddit source

Model Architecture: The Triple Role of a Minimalist Design

All Learnable Parameters

The entire model contains only four types of parameters:

A learnable 256-dimensional vector for each vocabulary token;
A learnable start state;
A "pull-strength" scalar;
A "readout-temperature" scalar.

The most crucial design is this: each token vector simultaneously plays three roles—it is the word's representation, the point-attractor, and the geometric readout. This means there is no additional decoder and no separate output matrix; all functionality is compressed into this shared "well table."

Trajectory as Encoding: How the State Vector Perceives Word Order

To encode a context, the system lets the state vector sequentially "pass through" ordered tokens. The core update formula is:

h ← h − strength · (1 − cos(h, W)) · normalize(h − W)

This formula has a clear geometric interpretation: (1 − cos(h, W)) is a monotonic function of angular distance—zero when h and W point in exactly the same direction (no pull), and reaching its maximum value of 2 when they point in exactly opposite directions (strongest pull); normalize(h − W) gives the unit direction vector from the current state toward the attractor. The overall update is therefore a "step along the connecting line, weighted by angular distance"—strong pull when far, weak pull when near, exhibiting a natural soft-constraint property.

The "point-attractor" here borrows a classic concept from dynamical systems: in phase space, an attractor is an invariant set toward which nearby trajectories converge over time. A point-attractor is the simplest type, corresponding to a stable equilibrium of the system, where all initial states within its basin of attraction eventually converge to the same point (more complex types include limit cycles and chaotic attractors). Each word vector defines a "gravity well" in the 256-dimensional space; as the state vector moves along the context sequence, it is successively tugged by each word's gravity well—a profound analogy to a point mass moving in a potential field in physics.

Notably, the author honestly points out that the model's "chord-directed force" is not a conservative force field, and there exists no exact global scalar potential, so it fundamentally differs from strict Hamiltonian dynamics. This non-conservativeness actually endows the system with path dependence on word order, so that the trajectory's endpoint carries sequence information. Because this is a trajectory, changing the token order changes the endpoint—naturally giving the model sensitivity to word order, which is exactly what distinguishes it from simple vector averaging (like the original CBOW summation).

Training Method and Hardware Barriers

CBOW-Style Fill-in-the-Blank Training

The training approach borrows the cloze-style idea from CBOW. CBOW (Continuous Bag of Words) is one of the classic training architectures in word2vec, proposed by Mikolov et al. in 2013, whose core idea is predicting the center word given a context window. Traditional CBOW simply averages context word vectors, inherently discarding word order information—the Collapse model precisely compensates for this deficiency through dynamical trajectories.

Specifically: for each qualifying noun token, the model reads the context of 5 tokens before and after it (a ±5 window) and requires the state to eventually settle near the "well" corresponding to the missing noun. At prediction time, it directly compares against these wells using cosine similarity, without any separate decoder.

The training scale is not small:

About 5 million lines of English Wikipedia corpus, roughly 300 million tokens;
94.75 million WordNet noun candidate token occurrences;
100,000 context vocabulary, 23,758 noun targets;
About 25.6 million parameters, almost entirely concentrated in the shared well table;
Training completed in a single streaming pass.

Most impressive is the hardware barrier: the entire training was completed on a single Apple Silicon MacBook via MPS, taking about 3.2 hours. Apple Silicon is Apple's series of ARM-architecture in-house chips (M1/M2/M3/M4) launched since 2020; its Unified Memory Architecture (UMA) lets the CPU, GPU, and Neural Engine share the same memory pool, eliminating the CPU-GPU data transfer bottleneck in traditional heterogeneous computing. MPS (Metal Performance Shaders) is Apple's low-level framework for GPU computing, and PyTorch has officially supported the MPS backend since version 1.12. Before this, individual researchers without an NVIDIA GPU often faced high cloud computing costs or long CPU training waits. This is extremely friendly to individual researchers' reproducibility, echoing the community ideal of "democratizing AI research" in recent years—it implicitly offers a critique of academia's growing reliance on ultra-large-scale computation.

Inference Speed

The encoder's inference speed is equally excellent: a single tokenized 10-token context takes about 0.23 milliseconds on CPU; MPS batch throughput reaches about 2.3 million tokens/second at batch size 1024; nearest-neighbor queries among the 23,758 nouns take about 0.48 milliseconds. This lightweight nature makes it genuinely attractive in resource-constrained scenarios.

What Semantic Structures Did the Model Learn

From nearest-neighbor examples, we can see the model did capture reasonable semantic structure:

physics → chemistry, mathematics, astronomy, quantum, mechanics
cat → tabby, dog, pet, felis, mouse, stray, feline
pakistan → karachi, punjab, lahore, peshawar, bangladesh, india
apple → macintosh, ipod, blackberry, android, pc, cherry

These results show that dynamics driven solely by co-occurrence pressure can spontaneously organize discipline clusters, animal categories, geographic associations, and even brand contexts. The "apple" result is particularly illustrative: the model simultaneously associates tech words like macintosh and ipod (the brand sense) while also surfacing cherry, the fruit sense—this phenomenon of "dominant sense primary, secondary sense residual" precisely reflects the inherent limitation of static single-vector representations for polysemous words: each word can only occupy one position and can only encode its "average semantic center of gravity" across the entire training corpus.

Honest Limitations

The author's candor about the model's flaws is commendable:

This is similarity, not reasoning or factual recall—don't expect it to do logic or knowledge-based Q&A.
A single vector per word can only capture the dominant sense, with limited ability to handle polysemy.

The polysemy problem is a long-standing and fundamental limitation of static word embeddings in representation learning. Within the word2vec and GloVe frameworks, the bank (financial) sense and bank (riverside) sense of the word "bank" are forced into the same vector, whose final position depends on the frequency-weighted average of the two senses in the training corpus. This limitation directly gave rise to research on contextual word representations: CoVe (McCann et al., 2017) uses a machine translation encoder to generate context vectors; ELMo (Peters et al., 2018) captures polysemy through hierarchical representations from a bidirectional LSTM; BERT (Devlin et al., 2019) makes each word's representation entirely dependent on the full sentence context in which it appears, via the Transformer's self-attention mechanism. Introducing polysemy within a geometric dynamics framework is an open problem—the author's proposed "multi-well" scheme, i.e., setting up multiple context-dependent attractors for each word, is conceptually similar to generalizing the single-Gaussian assumption into a Gaussian Mixture Model (GMM). But how to achieve dynamic well selection without introducing a full neural routing network is the core tension bridging minimalism and expressiveness, and the central challenge of this research direction.

Other limitations include:

WordNet noun determination is based on a dictionary, not contextual POS tagging.
A whole-word vocabulary means it cannot generalize to out-of-vocabulary (OOV) words.
The current "chord-directed force," although pointing toward the attractor, is non-conservative, with no exact global scalar potential—the author chose to document this correction rather than conceal it.
Comparisons with published word2vec/GloVe numbers are merely "suggestive" rather than controlled comparisons. The author has built a matched-corpus test framework with unified preprocessing, dimensions, vocabulary, data budget, and evaluation, to compare Collapse, SGNS, and PPMI+SVD.

SGNS (Skip-gram with Negative Sampling) is the most widely used variant in word2vec, introducing negative sampling to replace softmax computation and greatly improving training efficiency; PPMI+SVD is a traditional method based on co-occurrence matrices, applying truncated singular value decomposition to the Positive Pointwise Mutual Information matrix, and rivals neural methods on many semantic tasks. Notably, the theoretical work of Levy and Goldberg (2014) revealed a deep equivalence between SGNS and PPMI+SVD: SGNS implicitly performs a low-rank approximation of a shifted PMI matrix, unifying the two research traditions of neural word embeddings and distributional semantics. Against this backdrop, if the dynamical trajectory method can approach or even rival the performance of implicit matrix factorization methods, it would mean the semantic information in co-occurrence statistics can be equivalently extracted through multiple mathematical paths—a controlled comparison of Collapse against these two classes of methods is a necessary step to verify its practical value.

Additionally, the released v1 checkpoint predates a false-negative mask correction in the sampled softmax; its evaluation results are unaffected, but a corrected v2 retraining is still underway.

Significance and Open Questions

The author clearly states that the goal is not to claim it will replace standard embedding methods. What is truly exciting is that an ordered point-attractor dynamical system, without any conventional encoder network, can learn a useful semantic geometry from raw co-occurrence pressure. This provides a fresh geometric-dynamics perspective on the problem of "learning representations"—it suggests that semantic structure may not be an emergent phenomenon exclusive to neural networks, but a property that any ordered system capable of stably encoding co-occurrence statistics might spontaneously form.

The author poses two concrete questions to the community: first, beyond SGNS and PPMI+SVD, what fairer matched-data baselines exist; and second, how to introduce polysemy in a geometry-native way—i.e., setting up multiple context-dependent wells per word without introducing a full neural routing network.

The code and research notes are open-sourced on GitHub (chetanxpatil/livnium), and the model with a standalone loader is published on Hugging Face (chetanxpatil/noun-collapse). For researchers interested in the nature of representation learning, or who wish to explore minimalist architectures, this is an experimental sample worth trying.