Deep Dive into Word Embeddings: The Semantic Foundation of Large Language Models

Introduction: Why Embeddings Matter So Much

In the world of Large Language Models (LLMs), embeddings are a concept that is both fundamental and frequently misunderstood. This exploration, drawn from Chapter 2 of Hands-on Prep for Language Models (HPLM), has a title that candidly expresses the author's confusion and epiphanies while studying embeddings — "In Which I Lose My Mind over Embeddings."

Embeddings are critical because they serve as the bridge connecting human language to machine computation. Computers cannot directly understand textual symbols like "cat" or "dog" — they can only process numbers. The core mission of embeddings is to convert discrete vocabulary, characters, or subword units (tokens) into continuous high-dimensional vectors, enabling semantic relationships to be expressed and computed mathematically. This mapping from discrete symbols to continuous space is essentially the idea of Distributed Representation — each concept is no longer identified by a single symbol, but described collectively by continuous values across multiple dimensions. This allows models to capture subtle semantic gradients and multi-layered relational structures between words.

What Are Word Embeddings: From Symbols to Vectors

The Limitations of One-Hot Encoding

Traditional text representation methods like One-Hot Encoding suffer from two fatal flaws: first, dimensional explosion — the vector length equals the vocabulary size; second, semantic loss — any two word vectors are orthogonal, making it impossible to capture the relationship between "king" and "queen."

Specifically, one-hot encoding assigns each word in the vocabulary a unique index, then represents the word with a vector that has a 1 only at that index position and 0s everywhere else. For example, if the vocabulary contains 50,000 words, each word requires a 50,000-dimensional vector for representation, with only one meaningful dimension. This not only introduces enormous storage overhead, but more critically, the dot product of any two different one-hot vectors is always 0 (i.e., orthogonal). This means that mathematically, the distance between "good" and "excellent" is identical to the distance between "good" and "bad." In practical NLP tasks, this sparse high-dimensional representation also causes severe computational efficiency problems — the massive multiply-by-zero operations in matrix computations waste enormous computing power, while also making it difficult for models to learn effective generalization from limited training data.

Embeddings solve these problems. They map each token into a dense vector space of fixed dimensions (such as 768 or 4096 dimensions). In this space, semantically similar words are close to each other, and semantic relationships can even be captured through vector arithmetic. The classic example: vec(king) - vec(man) + vec(woman) ≈ vec(queen). This phenomenon is called linear analogy relationships, indicating that directions in the embedding space encode specific semantic relationships — the direction from "male to female," from "singular to plural," or from "country to capital" can all be captured by vector operations.

The Essence of the Embedding Matrix

From an implementation perspective, the embedding layer is essentially a large learnable matrix. If the vocabulary size is V and the embedding dimension is D, then the embedding matrix has dimensions V×D. When the model receives a token ID, it simply looks up the corresponding row vector in this matrix. This lookup process seems simple, but every value in the matrix has been continuously optimized through training on massive text corpora.

From a training mechanism perspective, every parameter in the embedding matrix is updated via gradient descent through backpropagation. When the model incurs a loss on a downstream task (such as next-token prediction), gradients propagate back to the embedding layer, adjusting the values in the activated row vectors. Notably, during each forward pass, only the row vectors corresponding to tokens in the current batch are updated, while the rest remain unchanged — this means that embeddings for low-frequency words are often lower quality than those for high-frequency words. In modern large models, the scale of embedding parameters is substantial: taking GPT-3 as an example, its vocabulary size is approximately 50,257 with an embedding dimension of 12,288, meaning the embedding matrix alone contains about 617 million parameters, accounting for roughly 0.35% of the total parameter count (175 billion). In smaller models like BERT-base, with a vocabulary of 30,522 and embedding dimension of 768, the embedding matrix has about 23 million parameters, accounting for roughly 21% of the total parameters (110 million). This shows that the larger the model, the smaller the relative proportion of the embedding layer, though its absolute scale remains significant.

The Core Difference Between Static and Contextual Embeddings

The Evolution from Word2Vec to Transformers

Much of the confusion learners experience when studying embeddings stems from the distinction between static and contextual embeddings. Early methods like Word2Vec and GloVe produce static embeddings — regardless of the context in which "apple" appears, its vector remains the same. This leads to serious ambiguity problems: Apple the tech company and apple the fruit share the same representation.

Reviewing the development of static embeddings helps understand this evolution. Word2Vec (proposed in 2013 by Google's Mikolov et al.) includes two training architectures: CBOW (Continuous Bag of Words) predicts the center word from context words, while Skip-gram does the reverse, predicting context words from the center word. Both are based on the distributional hypothesis — "a word's meaning is determined by its surrounding words." Skip-gram performs better with low-frequency words, while CBOW trains faster. GloVe (Global Vectors, proposed in 2014 by Stanford's Pennington et al.) took a different path: it first constructs a global word-word co-occurrence matrix, counting how frequently word pairs appear together across the entire corpus, then learns word vectors through matrix factorization so that the dot product of two word vectors equals the logarithm of their co-occurrence probability. This approach combines the advantages of global statistical methods and local context methods.

Between static embeddings and fully dynamic Transformer embeddings, there was an important transitional approach — ELMo (Embeddings from Language Models, proposed in 2018 by AllenNLP). ELMo uses a bidirectional LSTM language model and combines the hidden states from different layers as weighted combinations for word representations. It was the first to give "apple" different vector representations in different contexts, and is considered the pioneer of contextual embeddings. However, ELMo's limitation was that it could only capture sequence-level contextual dependencies, and the bidirectional LSTM's parallelization capability was far inferior to the later Transformer architecture.

In modern Transformer architectures, embeddings are just the starting point. Through the self-attention mechanism, each token's representation is dynamically adjusted based on context. Therefore, the same word receives different final representations in different sentences. Understanding this hierarchical relationship — from the initial embedding obtained through lookup, to the contextual representation processed through multiple network layers — is precisely the root cause of many learners' frustration. In Transformers, the vectors output by the initial embedding layer are still "static" (the same token ID always produces the same vector), but after processing through 12, 24, or even more layers of self-attention and feed-forward networks, the final hidden states have incorporated information from the entire sentence or even the entire context window, becoming truly context-dependent representations.

How Positional Encoding Merges with Word Embeddings

Another easily confused point is Positional Encoding. Since the self-attention mechanism itself is insensitive to order, the model needs an additional mechanism to inform each token of its position in the sequence. Positional information is typically added to or otherwise merged with word embeddings, making the boundary of the "embedding" concept blurry — it no longer represents only word meaning but also carries sequential information.

In the original Transformer paper ("Attention Is All You Need," 2017), the authors used sinusoidal positional encoding: for position pos and dimension i, encoding values are given alternately by sin(pos/10000^(2i/d)) and cos(pos/10000^(2i/d)). The elegance of this design lies in the fact that sinusoidal functions of different frequencies allow the model to represent relative positional relationships through linear transformations, while naturally extrapolating to sequence lengths unseen during training.

But this isn't the only option. Learnable Positional Embeddings is the approach adopted by models like BERT and GPT-2, which assigns a trainable vector to each position and learns optimal positional representations from data through backpropagation. The downside is that it cannot handle sequences exceeding the training length. In recent years, the most notable approach is RoPE (Rotary Position Embedding), proposed by Su Jianlin in 2021 and widely adopted by models like LLaMA and GPT-NeoX. RoPE's core idea is to encode positional information as rotation operations in vector space — relative position differences are expressed through differences in rotation angles. Its advantages include: naturally encoding relative positional relationships, good length extrapolation capability, high computational efficiency, and natural compatibility with the dot-product operations in self-attention.

Practical Applications of Embeddings in RAG and Semantic Search

From Keyword Matching to Semantic Search

The value of embeddings extends far beyond model internals. In practical applications, sentence-level or document-level embeddings are widely used for semantic search. By computing the cosine similarity between query vectors and document vectors, systems can find the most semantically relevant content rather than relying solely on keyword matching.

This semantic search implementation relies on Vector Databases such as Pinecone, Weaviate, Milvus, Qdrant, and Chroma. The core challenge for vector databases is: quickly finding the K most similar results to a query vector among millions or even billions of high-dimensional vectors. Exact nearest neighbor search (Exact KNN) requires traversing all vectors with O(n) time complexity, which is completely infeasible at large scale. Therefore, engineering practice universally adopts Approximate Nearest Neighbor (ANN) algorithms, such as HNSW (Hierarchical Navigable Small World, graph-based hierarchical search), IVF (Inverted File Index, which partitions the vector space into clusters and only searches relevant ones), and PQ (Product Quantization, which reduces memory and computational overhead through vector compression). These algorithms trade off between search accuracy and speed, typically retrieving results with over 95% accuracy from billion-scale data within milliseconds.

In terms of embedding model selection, current mainstream options include: OpenAI's text-embedding-3-large (3072 dimensions, supporting flexible dimensionality reduction), Cohere's Embed v3, and open-source models from the BGE, E5, and GTE series. When choosing an embedding model, considerations include dimension size (affecting storage cost and retrieval speed), maximum supported sequence length (affecting document chunking strategy), and performance on domain-specific benchmarks (such as the MTEB leaderboard).

This is precisely the foundation of the currently hot RAG (Retrieval-Augmented Generation) technology. Converting knowledge base documents into embeddings and storing them in a vector database, then retrieving relevant fragments when users ask questions and injecting them into the prompt, can significantly improve the accuracy and timeliness of LLM responses. The typical RAG pipeline includes: Document Chunking → Embedding Generation → Vector Storage → Query Embedding → Similarity Search → Context Injection → LLM Answer Generation. Among these, the document chunking strategy (by fixed length, by semantic paragraph, or by recursive character splitting) and embedding model selection are key decisions that directly affect retrieval quality.

The Right Path to Learning Embeddings

For developers seeking to deeply understand LLMs, the author's experience offers useful insights: don't try to understand all the details at once, but proceed step by step. First master the intuition behind static embeddings, then understand the dynamic nature of contextual embeddings, and finally grasp the role of auxiliary mechanisms like positional encoding. Hands-on practice — such as visualizing embedding spaces or computing word vector similarities — often brings more "eureka moments" than pure theoretical reading.

For visualization practice, t-SNE (t-Distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection) are the two most commonly used tools for dimensionality reduction and visualization of high-dimensional vectors. t-SNE excels at preserving local structure in data, clearly showing clustering relationships, but is computationally slow and doesn't preserve global distance information. UMAP better maintains global topological relationships while preserving local structure, is significantly faster, and is more suitable for large-scale datasets. Practical recommendations include: use Python's sentence-transformers library to obtain sentence embeddings, then use sklearn's t-SNE or the umap-learn library for dimensionality reduction, and finally visualize with matplotlib or plotly. By comparing the embedding distributions of "tech news" versus "sports news," or observing how the same polysemous word shifts position in different contexts, developers can intuitively understand the geometric structure and semantic organization of embedding spaces. Additionally, TensorBoard's Embedding Projector is a very intuitive interactive visualization option.

Conclusion

Embeddings are an indispensable foundation for understanding modern language models. They seem simple — just a lookup operation — yet they embody profound representation learning principles. As the title of HPLM Chapter 2 suggests, the process of deeply studying embeddings is full of challenges and confusion, but it is precisely this "mind-losing" exploration that builds a solid understanding of how LLMs work. For every AI learner, crossing the threshold of embeddings is what truly opens the door to the world of large language models.