Embedding Model

Embedding models typically map text into high-dimensional vector spaces of 768 or 1536 dimensions.

Embedding models convert text into fixed-dimensional floating-point vectors, placing semantically similar content closer together in the vector space, and serve as the core foundation of RAG technology.

RAG通过将文档切分为语义块后经嵌入模型转化为高维向量存入向量数据库，查询时通过近似最近邻（ANN）搜索找到语义最相近的文档片段

Text is encoded by an Embedding model into floating-point vectors of 768 to 3072 dimensions, stored in a vector database, and similarity retrieval is completed at millisecond speed via approximate nearest neighbor algorithms (HNSW, IVF, etc.).

Embedding models are neural networks that map text to a continuous vector space, with the core assumption that semantically similar texts should have vectors that are close in distance.

Vectorizing documents and vectorizing user queries must use the same embedding model; otherwise, incompatible vector spaces render similarity calculations meaningless.

Embedding models and generative models are two different types of models: the former outputs fixed-dimension vectors, while the latter outputs text.

No matter how complex any RAG system is, it essentially has only two steps: the first step is to fill the library (load documents, split into chunks, embed, and store in a vector database), and the second step is to retrieve information (convert the user's question into a vector, retrieve the closest chunks, and hand them together with the original question to the LLM to generate an answer).

In a RAG system, the two steps of building the library and retrieval must use the same Embedding model, otherwise the two models have different definitions of similarity, the vector spaces cannot be aligned, and the retrieval results will deviate severely.

Embedding models map queries and documents into the same vector space and match them using measures such as cosine similarity; when a query uses domain abbreviations, internal codenames, or wording whose distribution differs greatly from the training corpus, the correct document may be ranked very far down or may not even appear in the candidate set.

The solution described in this article emphasizes that both the language model and the vector model are deployed locally, rather than calling external cloud-based APIs.

The preparation stage for building a vector database requires splitting private documents into smaller chunks, converting them into vectors via an Embedding model, and then storing them in the vector database.

In the RAG retrieval process, the user's question is first passed to an Embedding model to be converted into a vector, then similarity calculations are performed in a vector database to retrieve the top-K most relevant documents, which are then concatenated with the original question and passed to the large model to generate an answer.

The vector dimensions output by different embedding models range from 768 to 3072; higher dimensions generally provide stronger semantic representation capability but incur greater storage and computation overhead

The Embedding model appears twice in the RAG pipeline: during offline knowledge base construction and during online retrieval, with the goal in both cases being to map text into the same vector space.

During model selection, it is necessary to confirm that the similarity objective function used during model training is consistent with the metric used during retrieval; otherwise, systematic bias will occur.

Embedding model fine-tuning typically adopts a contrastive learning framework, training the model to bring positive sample pairs closer together and push negative sample pairs apart by constructing positive and negative sample pairs.

The core function of an Embedding model is to convert a piece of plain text into a vector.

The Embedding model is only responsible for the text-to-vector step; which database the vectors are stored in is a relatively independent engineering choice.

Embedding models lack special understanding of structured symbols (such as currency formats, encoded strings, and ID sequences), treating this content merely as ordinary token sequences.