Word Embeddings Explained: The Logical Relationship Between Word2Vec, SVD, and GloVe

A Confused NLP Beginner

I recently saw a student taking an NLP course post a soul-searching question on Reddit: The professor covered one-hot vectors, embedding matrices, Word2Vec, co-occurrence matrices with SVD, GloVe… but what are all these things actually for? Why, if we already have dense vectors, do we need co-occurrence matrices and SVD to generate word embeddings? Why do we need to compute ratios of conditional probabilities?

This kind of confusion is actually very common. Many courses throw all the various word embedding methods at students at once without clearly explaining the logical relationships and shared goal between them. This article will untangle this thread and help you connect these seemingly scattered concepts into a coherent narrative.

The Core Problem: How to Represent Word Semantics with Vectors

All of these methods are essentially solving the same problem: How to represent a word as a numerical vector such that semantically similar words are close to each other in the vector space.

The theoretical foundation for this problem comes from the Distributional Hypothesis in linguistics, summarized by J.R. Firth's famous 1957 dictum: "You shall know a word by the company it keeps." In other words, if two words consistently appear in similar contexts, their meanings should be similar. This hypothesis forms the philosophical foundation of virtually all word embedding methods—whether prediction-based or count-based, they're all attempting to capture "contextual similarity" through mathematical means.

The Limitations of One-Hot Vectors

The most naive approach is one-hot encoding: assuming a vocabulary of 50,000 words, each word becomes a 50,000-dimensional vector with a 1 at its corresponding position and 0s everywhere else.

This representation has two fatal flaws:

• Curse of dimensionality: The vector dimension equals the vocabulary size—extremely sparse and enormous. In practice, a 50,000-word vocabulary means each word requires 50,000 floating-point numbers of storage, with 99.998% of positions being 0. When vocabularies scale to hundreds of thousands (like an English Wikipedia vocabulary), memory consumption and computational efficiency become unacceptable.
• Cannot express semantics: The dot product of any two one-hot vectors is 0, meaning the "distance" between "cat" and "dog" is exactly the same as between "cat" and "car"—the model cannot perceive any similarity. Mathematically, one-hot vectors are mutually orthogonal; the space they form encodes no structural information about word meaning.

So we need to "compress" one-hot vectors into low-dimensional dense vectors—this is exactly what the embedding matrix does. Multiplying a one-hot vector by an embedding matrix essentially looks up the corresponding row. Specifically, if the embedding matrix has dimensions V×d (V is vocabulary size, d is embedding dimension, typically 100-300), then a 1×V one-hot vector multiplied by it yields the i-th row of the matrix—a d-dimensional dense vector. In deep learning frameworks, this operation is typically implemented as an efficient "lookup" operation, avoiding actual matrix multiplication.

Key Insight: The Embedding Matrix Must Be Learned Through Training

The questioner's biggest confusion was: "If we already have dense vectors, why do we need Word2Vec, SVD, and GloVe?"

The answer is: These methods are precisely what we use to learn that embedding matrix. Dense vectors don't exist out of thin air—every value in the matrix needs to be obtained through training. Word2Vec, SVD, and GloVe are simply three different "training/solving" approaches. Once you understand this point, the entire narrative clicks into place.

Another way to think about it: when an embedding matrix is first created, its values are randomly initialized and contain no semantic information. Only through some learning process—where the values are continuously adjusted based on statistical patterns in a corpus—can these vectors "acquire" semantics. Different word embedding methods are essentially different "adjustment strategies."

Comparing the Three Technical Approaches to Word Embeddings

Word embedding methods roughly divide into two major schools: prediction-based methods (Word2Vec) and count-based methods (co-occurrence matrix + SVD, GloVe).

Approach 1: Word2Vec — Prediction-Based Word Embeddings

Word2Vec was proposed by Tomas Mikolov et al. at Google in 2013 and is a landmark work that brought word embeddings into mainstream applications. Its core idea is: have the model "predict" context. Word2Vec contains two architectures: Skip-gram and CBOW (Continuous Bag of Words). Skip-gram predicts context words given a center word, while CBOW does the reverse—predicting the center word from context words. Both achieve similar results, but Skip-gram typically performs better on rare words, while CBOW trains faster.

Taking Skip-gram as an example, given a center word, it predicts the words appearing within a surrounding window. The specific process is:

1. Take the dot product of the dense vectors for the center word and context word to get a score;
2. Convert the score to a probability using softmax;
3. Construct the log-likelihood from the probability;
4. Minimize the negative average log-likelihood (i.e., the loss function).

So "then what? So what?"—this is the key point. During loss minimization, the model continuously adjusts the values of those dense vectors. After training is complete, those repeatedly optimized vectors are the word embeddings we want. In other words, Word2Vec's "byproduct"—that intermediate vector table—is what we actually want. Words appearing in similar contexts get trained to have similar vectors.

It's worth noting that in practice, computing softmax over the entire vocabulary is too expensive (requiring exponentiation and normalization over tens of thousands of words), so Negative Sampling is used: instead of computing the full softmax, the problem is reformulated as binary classification—for each (center word, context word) pair, only a few randomly sampled "noise words" serve as negative examples, and the model learns to distinguish real context words from random ones. This reduces per-step complexity from O(V) to O(k), where k is the number of negative samples (typically 5-20), making training on large-scale corpora feasible.

After training, Word2Vec produces a famous property: arithmetic operations between vectors can capture semantic relationships, for example vec("king") - vec("man") + vec("woman") ≈ vec("queen"). This demonstrates that the learned vector space possesses structured semantic geometric properties.

Approach 2: Co-occurrence Matrix + SVD — Count-Based Word Embeddings

The count-based approach takes a completely different philosophy. It first counts how often every word co-occurs with every other word within a window across the entire corpus, building a massive co-occurrence matrix.

The academic lineage of this approach traces back to LSA (Latent Semantic Analysis), proposed by Deerwester et al. in 1990. LSA originally applied SVD to a term-document matrix for information retrieval. Researchers later discovered that applying SVD to a word-word co-occurrence matrix works better, more directly capturing semantic relationships between words.

The core intuition is: if two words frequently appear in similar contexts (similar co-occurrence patterns), their semantics are similar. For example, "doctor" and "nurse" both frequently co-occur with "hospital," "patient," and "treatment," so their corresponding row vectors in the co-occurrence matrix will be very similar.

But this co-occurrence matrix is also high-dimensional and sparse (a V×V square matrix, where V could be tens to hundreds of thousands), so SVD (Singular Value Decomposition) is used to reduce its dimensionality, retaining only the most important dimensions to produce low-dimensional dense word vectors. The mathematical intuition behind SVD is: decompose the original matrix into a product of three matrices M = UΣV^T, where Σ is a diagonal matrix with singular values arranged by magnitude. Retaining only the top d largest singular values (and corresponding columns in U and V) yields the best d-dimensional approximation of the original matrix. Each row of the U matrix can then serve as the d-dimensional embedding vector for the corresponding word.

In practice, researchers found that applying SVD directly to the raw count matrix doesn't work well because high-frequency word pairs (like "the" co-occurring with almost everything) dominate the results. A better approach is to first apply a PMI (Pointwise Mutual Information) transformation to the co-occurrence matrix—PMI(w,c) = log[P(w,c)/(P(w)·P(c))]—and then perform SVD. PMI effectively assigns high weight to "surprising co-occurrences" while suppressing noise from pure high frequency. Levy and Goldberg's 2014 work proved that Word2Vec's Skip-gram with negative sampling is actually implicitly factorizing a PMI matrix, thereby revealing a deep mathematical connection between the prediction and count-based camps.

So to answer the questioner's concern: SVD here isn't "generating vectors a second time"—it's an entirely independent way of generating word embeddings. It exists in parallel with Word2Vec, not in sequence. The course presents them together to contrast the "prediction" and "count" philosophies.

Approach 3: GloVe — Merging Prediction and Count-Based Approaches

GloVe (Global Vectors for Word Representation) was proposed by Jeffrey Pennington, Richard Socher, and Christopher Manning at Stanford in 2014. It attempts to combine the strengths of both: leveraging global corpus statistics (the count-based camp's strength) while maintaining the flexibility of vector learning (the prediction-based camp's strength).

Its core insight is: Semantic relationships between words are encoded in the ratios of co-occurrence probabilities.

A classic example: for "ice" and "steam,"

• "solid" co-occurs with ice far more than with steam, so the ratio P(solid|ice)/P(solid|steam) is large (≈8.9);
• "gas" co-occurs more with steam, so the ratio P(gas|ice)/P(gas|steam) is small (≈0.085);
• "water" is related to both, so the ratio is close to 1 (≈1.36);
• "fashion" is unrelated to both, so the ratio is also close to 1 (≈0.96).

This ratio precisely encodes the discriminative semantics between words—it naturally eliminates interference from "noise words" related to both or neither word, retaining only the discriminative signal.

GloVe's loss function is designed so that the dot product of word vectors w_i and w_j approximates the logarithm of their co-occurrence count: w_i^T · w_j + b_i + b_j ≈ log(X_ij), where X_ij is the co-occurrence count. It also introduces a weighting function f(X_ij) that caps extremely high-frequency co-occurrence pairs (typically truncating at X=100), preventing high-frequency words like "the" from dominating the loss; zero co-occurrences don't participate in training. This design allows GloVe to leverage global statistical information (count-based advantage) while preserving vector arithmetic capabilities (prediction-based advantage), and training only needs to iterate over non-zero co-occurrence pairs, making it more efficient than full-matrix SVD.

Why NLP Learners Need to Master Multiple Methods

Now back to the original confusion: if the goal is always to get word embeddings, why learn so many methods?

• They represent historical evolution and intellectual contrast. From SVD to Word2Vec to GloVe, they reflect the NLP field's deepening understanding of "how to represent word meaning." This progression traces from purely mathematical methods (matrix factorization) to end-to-end neural network learning (prediction models) to theoretical unification (proving their equivalence).
• They have different trade-offs. Word2Vec is training-efficient, easily scalable, and suitable for incremental training on large corpora; SVD makes fuller use of global statistics but has O(V³) computational complexity for exact SVD on a V×V matrix, which is costly for large vocabularies; GloVe attempts to combine the best of both and performs on par with or better than Word2Vec on various benchmarks. Understanding the differences enables informed choices in practice.
• They are stepping stones to modern models. Today's BERT, GPT, and other large models have long surpassed static word embeddings, replacing fixed vectors with context-dependent representations. But their underlying intuition—"represent semantics with vectors, learn representations through prediction tasks"—grew directly from methods like Word2Vec.

Regarding the evolution from static to dynamic representations, an important transitional milestone worth mentioning is ELMo (Embeddings from Language Models), proposed by Allen AI in 2018. ELMo uses a bidirectional LSTM language model to generate different vectors for each word depending on its sentence context. For example, "bank" in "river bank" and "bank account" receives different representations. This solved a fundamental limitation of Word2Vec/GloVe—static word embeddings cannot handle polysemy. ELMo bridged the gap from static word embeddings to BERT-style Transformer contextual representations, helping us understand why the "pretrain representations + downstream fine-tuning" paradigm is so powerful. From Word2Vec's "predict context" to GPT's "autoregressive language model," the core idea remains consistent: leveraging distributional patterns in large-scale text to learn general-purpose language representations.

A Unified Framework for Beginners

If you're also getting dizzy from all these concepts, keep this unified framework in mind:

All word embedding methods do one thing—learn a table that maps words to dense vectors, making semantically similar words close in space. They differ only in "what signal" and "what method" they use to learn this table.

• Word2Vec uses local prediction signals (context words within a window), learned via gradient descent;
• Co-occurrence matrix + SVD uses global count signals (co-occurrence statistics across the entire corpus), learned via matrix factorization;
• GloVe uses global count ratio signals (ratio relationships of co-occurrence probabilities), learned by fitting a loss function.

From the perspective of learning signals, Word2Vec is "local window, online learning"—it sees one window at a time and gradually updates via stochastic gradient descent; SVD and GloVe are "global statistics, batch learning"—they first aggregate information from the entire corpus, then solve in one pass or iteratively. Research after 2014 (particularly Levy & Goldberg's analysis) showed that these methods are mathematically more unified than they appear on the surface: they all factorize some form of matrix containing co-occurrence information, just using different objective functions and optimization strategies.

Once you grasp this main thread, those seemingly isolated formulas and procedures will all fall into place. The process of learning NLP is exactly the process of continuously reducing scattered techniques back to a unified goal.