A Deep Dive into Vector Search for AI Agents: The Technical Foundations of RAG and Long-Term Memory

In today's era of rapidly advancing AI Agents, we often marvel at how large models not only answer questions but can also "remember" previous conversations, and even flip through internal enterprise documents to provide precise answers. What technology actually underpins these capabilities? The answer points to a single core concept—high-dimensional vectors and approximate search. Starting from the fundamental principles of the vector world, this article breaks down in plain language the three core algorithms that support AI Agent memory and knowledge retrieval: HNSW, IVF, and PQ, and ultimately reveals how they underpin the complete pipeline of RAG and long-term memory.

Everything Can Be Vectorized: How Computers Quantify the World

Humans perceive the world through their eyes, ears, and sense of touch, but a computer sees only numbers. How does it distinguish between an apple and an orange? This requires us to help it build a numerical understanding—which can be thought of as a "leap in dimensions."

Consider a scenario: we want a computer to distinguish between a mandarin orange and a navel orange. If we give it only one feature—weight—and both weigh 80 grams, they completely overlap on a one-dimensional axis, and the computer simply can't tell them apart. Adding a second feature, color (golden versus greenish), pulls them slightly apart in two-dimensional space; adding a third dimension, skin roughness, completely separates the two fruits.

It's worth emphasizing that weight, color, and roughness are features humans can intuitively interpret. But in actual engineering, the 128-dimensional or even thousands-of-dimensional vectors extracted by a large model's Embedding Model each represent complex latent semantic features that cannot be directly described in words.

Technical background of embedding models: An embedding model is a neural network that transforms unstructured data such as natural language, images, and audio into dense vectors. Its core idea originated from Word2Vec, proposed by Google in 2013, and has since evolved through models like BERT, Sentence-BERT, and OpenAI's text-embedding series, dramatically improving embedding quality. Modern embedding models are typically based on the Transformer architecture and are trained through large-scale Contrastive Learning, causing semantically similar content to cluster together in the vector space while semantically distant content is pushed apart. Typical embedding dimensions range from 128 to 4096; the higher the dimensionality, the richer the semantic detail captured, but storage and computational costs grow linearly as well.

However, the relative geometric positions they form in multidimensional space follow exactly the same principle as the orange example—the more dimensions, the more precise the depiction.

The Essence of Semantic Similarity Is Geometric Distance

Determining whether two things are similar essentially comes down to their geometric distance in high-dimensional space—the closer the distance, the more similar the semantics. For example, if we build a music-listening space using three dimensions—tempo, electronic music proportion, and vocal ratio—a lullaby and a soothing jazz track would sit very close together, while an energetic heavy metal song would be flung off to another corner of the space.

Therefore, any semantic search can be broken down into three steps: extract multimodal features → vectorize via an embedding model → compute geometric distance to pinpoint the nearest node. This leads to the classic "Nearest Neighbor" (NN) problem in computer science. When a database contains hundreds of millions of vectors, finding the closest one within a few milliseconds becomes a genuine engineering challenge.

HNSW: A Detailed Explanation of the Hierarchical Navigable Small World Graph Algorithm

Faced with billion-scale data, traditional database indexes (such as the B+ tree in SQL) fail outright because they cannot establish an absolute ordering in multidimensional space; brute-force search computes distances one by one, and a single query could take several minutes—far too slow for real-time interaction. HNSW (Hierarchical Navigable Small World) was born precisely to solve this problem.

Sounds pretty convoluted, right?

Breaking down the name: "Small World" borrows from sociology's Six Degrees of Separation theory—by introducing friend to friend, you can reach anyone in the world through at most six people.

The Six Degrees of Separation theory and the mathematical foundation of small-world networks: The Six Degrees of Separation theory was validated by social psychologist Stanley Milgram in 1967 through his "small-world experiment": any two strangers can, on average, be connected through just six intermediaries. Mathematicians Duncan Watts and Steven Strogatz formalized this in 1998 as the "small-world network" model, revealing two core characteristics of such networks: high clustering coefficient (local neighbors are interconnected) and short average path length (very few hops between any two nodes). HNSW transplants this topological property into vector retrieval—through a carefully designed edge-connection strategy, it can reach the vicinity of a target vector in O(log n) hops even at a scale of tens of billions of nodes, which is the mathematical foundation of its ultra-high retrieval efficiency.

In the vector space, we connect nearby vectors with edges to form a neighbor network. During retrieval, we randomly pick a starting node and jump along the network toward the target direction, quickly homing in on the location.

"Hierarchical" is HNSW's most ingenious design, much like a transportation system: the top layer is the "highway," with sparse nodes that allow large leaps to quickly lock onto a broad region; the middle layer is the "provincial and national roads," with denser nodes that continue narrowing the search circle; the bottom layer is the "city streets," containing the full dataset for precise localization.

There's a key underlying principle here—read-write separation. When inserting a new vector, you must perform "connection establishment" to alter the graph structure; when querying, you follow the same hierarchical pathfinding process, but the final step only reads and returns the ID of the nearest node—it never participates in forming connections. When learning HNSW, never confuse read and write operations.

HNSW can achieve near-logarithmic ultra-fast response and is the mainstream choice in industry. But the cost is equally obvious: the complex multi-layer neighbor relationship graph and all node pointers must reside entirely in physical memory. Once the data volume reaches hundreds of millions, server memory faces severe pressure.

IVF and PQ: Pushing Memory and Compute to the Limit

Since memory cannot hold the massive relationship network, we must optimize space at the algorithmic level. This is where the two key algorithms IVF and PQ come into play.

But there's no free lunch under the celestial sphere

IVF Inverted File Index: Drastically Narrowing the Search Scope

IVF stands for Inverted File Index. Normally we store data as "which features a given piece of data has," but the inverted approach is exactly the opposite—first partition the feature regions, then build a list recording which data IDs each region contains. It's like shopping in a mall: first locate the "men's clothing section," then go in and find the specific brand.

It's implemented in three steps in engineering: first, use K-Means clustering training to automatically group the vast amount of data into different "folders," each folder generating a representative "cluster center"; next, coarse filtering and localization—when query vector Q comes in, it's first compared with each cluster center to quickly lock onto the best-matching folder; finally, local fine search—distances are computed only within the locked folder. Irrelevant regions are excluded outright, cutting the computation by over 90%.

Engineering details of K-Means clustering: K-Means is the core preprocessing step for building the IVF index. The algorithm works as follows: randomly initialize K cluster centers, assign each data point to the nearest center, recompute the mean of each cluster as the new center, and iterate until convergence. In vector database scenarios, K is typically set to the square root of the data volume (e.g., about 1,000 clusters for one million data points) to balance the candidate set size after each coarse filter against the computational cost of comparing cluster centers. K-Means training is done offline. Once the index is built, online retrieval only requires comparing the query vector against the K cluster centers once, which compresses the search scope from the full dataset to 1/K—this is the core mechanism behind IVF's order-of-magnitude acceleration.

PQ Product Quantization: Pushing Vector Storage Size to the Limit

IVF narrows the search scope, but the vectors within each folder are still large in size. This is where the key technique of PQ (Product Quantization) comes in.

So how do we split and encode?

"Quantization" means finding an approximate representative, like compressing a high-definition photo into mosaic blocks—losing detail but keeping the outline; "product" can be understood as segmented combination—slicing a long vector into several segments, finding a representative for each, then combining them.

Take a four-dimensional vector as an example (simplified to integers 64, 91, and so on): originally, storing 4 floating-point numbers requires 16 bytes. In PQ's first step, splitting, we slice it into two two-dimensional sub-vectors; in the second step, local clustering, we find the nearest cluster center within each sub-space (e.g., the first half is close to center No. 2, the second half close to center No. 3); in the third step, encoding substitution, we directly store the codebook indices of the cluster centers (e.g., 2 and 3). Data that originally took 16 bytes can now be represented by two single-byte indices, compressing the physical size by a factor of 8.

The information-theoretic basis of product quantization: PQ's design is essentially a form of lossy data compression, with its theoretical basis rooted in Rate-Distortion Theory from information theory. It splits a high-dimensional vector into M sub-spaces, independently trains a Codebook of K* cluster centers for each sub-space, and ultimately replaces the original floating-point vector with M integer codes. Compared to clustering the entire vector directly, this "divide and conquer" quantization strategy exponentially expands the effective codeword space: with M sub-spaces each containing K* centers, it can theoretically express (K*)^M combinations while requiring only M×log₂(K*) bits of storage. For example, with M=8 and K*=256, the theoretical number of combinations reaches 256^8 ≈ 1.8×10¹⁹, yet storage requires only 8 bytes—this is precisely the mathematical root of PQ's ability to maintain relatively high approximation accuracy at minimal storage cost.

ADC Asymmetric Distance Computation: Replacing Floating-Point Operations with Table Lookups

Do we need to decompress and restore the compressed data before performing floating-point operations during retrieval? The answer is no. The industry adopts ADC (Asymmetric Distance Computation): the user's query vector Q is kept at its original high precision without compression. First, Q is used to compute the true distances to all sub-space cluster centers in the codebook, generating a tiny "distance Look-Up Table" (LUT). Afterward, when computing the distance between Q and a compressed vector, there's no need for multiplication or square roots at all—we simply look up the values from the table by index and perform simple additions to obtain the approximate distance. This "keep the query as-is, compress the database" design replaces energy-intensive floating-point operations with table lookups and additions, boosting retrieval efficiency exponentially.

The Art of Trade-offs: The Core Principle of ANN (Approximate Nearest Neighbor)

Each of the three algorithms has its trade-offs, and this is precisely where the art of engineering design lies: HNSW retrieves extremely fast but may fall into a local optimum and miss the true nearest neighbor; IVF drastically reduces computational overhead, but misses can occur when the target happens to fall on a folder boundary; PQ achieves high compression, but replacing true values with indices inevitably introduces precision loss. This is the core principle of ANN (Approximate Nearest Neighbor)—the essence of a vector database has never been to pursue 100% absolute precision, but to trade a tiny margin of error for hundreds or thousands of times greater efficiency in time and space.

From Algorithm to Application: RAG and AI Agent Long-Term Memory

Having grasped the underlying algorithms, let's look at how they support the practical capabilities of AI Agents.

These underlying technologies

First, we need to clarify a hierarchical relationship: an underlying algorithm library like Faiss is like a "technologically advanced car engine," but you can't just sit on the engine and drive.

The ecosystem layering of Faiss and vector databases: Faiss (Facebook AI Similarity Search) was open-sourced by Meta AI Research in 2017 and is currently the most widely used low-level vector retrieval library in industry. Written in C++ with a Python interface, it has built-in HNSW, IVF, PQ, and their combined indexes (such as IVF+PQ) and supports GPU acceleration. However, Faiss is essentially an algorithm library and lacks database capabilities such as persistent storage, multi-tenant permission management, and real-time create/update/delete operations. Built on top of it, vector databases such as Milvus (open-sourced by Zilliz, suited for ultra-large-scale distributed scenarios), Qdrant (written in Rust, emphasizing memory safety and low latency), Weaviate (with native multimodal retrieval support), and Pinecone (a fully managed cloud service, ready out of the box) each have their own focus, forming a complete ecosystem hierarchy. Selection typically considers three key dimensions: data scale (single-machine vs. distributed), update frequency (static index vs. real-time writes), and hybrid query needs (pure vector vs. vector + scalar filtering).

You also need a vector database like Milvus or Qdrant to provide system-level encapsulation—it packages and integrates engines like HNSW, IVF, and PQ, and provides enterprise-grade data management, hybrid queries with metadata filtering, and distributed high-availability guarantees. Only with this complete "vehicle" can AI applications run fast and stable.

RAG Retrieval-Augmented Generation: Giving the Large Model a Reference Book

RAG (Retrieval Augmented Generation) is like giving a large model an open-book exam reference book. The workflow is: the user asks a question → the system vectorizes the question into Q → uses the aforementioned algorithms to scan the vector database and lock onto the most relevant text passages → concatenates the passages with the question and injects them into the Prompt → the large model, empowered by accurate reference material, outputs a high-quality, hallucination-free answer. Throughout this process, the large model handles reasoning while the vector database handles retrieval, successfully transforming complex semantic lookup into pure geometric computation.

The technical evolution and limitations of RAG: The concept of RAG was formally proposed by Meta AI in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," initially for open-domain question answering. As the wave of large model applications rose, RAG quickly became the standard paradigm for deploying enterprise knowledge bases. RAG has now evolved into several variants: Naive RAG is the most basic two-stage "retrieve-generate" process; Advanced RAG introduces query rewriting, re-ranking, and context compression to improve recall quality; Modular RAG decouples each module to support flexible orchestration. RAG's core limitation lies in the "retrieval ceiling"—if the vectorization quality is insufficient or the knowledge base coverage is incomplete, then no matter how powerful the generation model is, the final answer quality is limited by the recall and precision of the retrieval stage. This is also the main direction of current research breakthroughs.

Long-Term Memory: A Dynamic Retrieval Loop That Lets AI "Remember" You

Vector retrieval is also the underlying pillar of an AI Agent's memory, forming an elegant dynamic loop. When we converse with an AI daily, historical interactions are converted into feature vectors in real time and persistently archived in the vector database; when a new round of conversation begins, the system generates a new Query vector to instantly awaken the most relevant historical memory fragments. This "memory archiving—memory awakening" process can recall historical information at the millisecond level, allowing the AI to "remember" previous agreements and deliver a continuous, personalized conversational experience.

Conclusion

No matter how varied the application-layer forms may be—whether it's RAG acting as an external knowledge base, or the long-term memory that gives AI Agents human-like interaction capabilities—tracing down the technical chain, they all ultimately point to the same endpoint: multidimensional vectors and approximate search. The cornerstone that endows machines with the ability to understand the world and capture human intent is forever the high-dimensional vector space that turns everything into numbers. Once you understand this, you have truly grasped how AI thinks.