Embedding Model

Embedding models typically map text into high-dimensional vector spaces of 768 or 1536 dimensions.

Embedding models convert text into fixed-dimensional floating-point vectors, placing semantically similar content closer together in the vector space, and serve as the core foundation of RAG technology.

RAG通过将文档切分为语义块后经嵌入模型转化为高维向量存入向量数据库，查询时通过近似最近邻（ANN）搜索找到语义最相近的文档片段

Embedding models are neural networks that map text to a continuous vector space, with the core assumption that semantically similar texts should have vectors that are close in distance.

Embedding models map queries and documents into the same vector space and match them using measures such as cosine similarity; when a query uses domain abbreviations, internal codenames, or wording whose distribution differs greatly from the training corpus, the correct document may be ranked very far down or may not even appear in the candidate set.

The solution described in this article emphasizes that both the language model and the vector model are deployed locally, rather than calling external cloud-based APIs.

The preparation stage for building a vector database requires splitting private documents into smaller chunks, converting them into vectors via an Embedding model, and then storing them in the vector database.

In the RAG retrieval process, the user's question is first passed to an Embedding model to be converted into a vector, then similarity calculations are performed in a vector database to retrieve the top-K most relevant documents, which are then concatenated with the original question and passed to the large model to generate an answer.

The vector dimensions output by different embedding models range from 768 to 3072; higher dimensions generally provide stronger semantic representation capability but incur greater storage and computation overhead

During model selection, it is necessary to confirm that the similarity objective function used during model training is consistent with the metric used during retrieval; otherwise, systematic bias will occur.

Embedding model fine-tuning typically adopts a contrastive learning framework, training the model to bring positive sample pairs closer together and push negative sample pairs apart by constructing positive and negative sample pairs.

The core function of an Embedding model is to convert a piece of plain text into a vector.

The Embedding model is only responsible for the text-to-vector step; which database the vectors are stored in is a relatively independent engineering choice.

The Embedding model appears twice in the RAG pipeline: during offline knowledge base construction and during online retrieval, with the goal in both cases being to map text into the same vector space.

Embedding models lack special understanding of structured symbols (such as currency formats, encoded strings, and ID sequences), treating this content merely as ordinary token sequences.

Structured data such as prices, dates, and categorical IDs should generally not be directly fed into embedding models, but should instead be stored as metadata for filtering and sorting after retrieval.

Directly embedding structured fields pollutes the semantic space, causing the model to be unable to distinguish prices from arbitrary text.

The direction of embedding vectors for overly short texts tends to be unstable, making filtering them out the safer choice.

Open-source embedding models such as BAAI/bge-large and intfloat/e5-large-v2 have approached or even surpassed the performance of commercial APIs on multiple benchmarks.