Milvus 2.6 in Practice: Building a High-Performance RAG Q&A System

An analysis of Milvus 2.6's cloud-native architecture and key optimization practices for high-performance RAG systems.
Based on a sharing by technical instructor Li Jin, this article introduces Milvus 2.6's core features and architecture starting from vector retrieval principles. It explains how vector databases solve high-dimensional retrieval of unstructured data via approximate nearest neighbor search, and compares Milvus's role in AI applications to MySQL in Java development. It then analyzes Milvus 2.6's cloud-native layered architecture—Stream, Query, and Data Nodes plus Coordinators, along with dependencies like etcd, Kafka, and object storage—and clarifies the relationship with the commercial Zilliz Cloud. Deployment covers Lite and standalone hardware requirements. The hands-on section highlights four optimization dimensions for high-performance RAG: watermark management, preload fields, index strategy, and tiered storage eviction. It concludes with a look at 2026's LLM technical focus and career intersections.
Vector databases are becoming the infrastructure for AI applications, and Milvus, as a leading open-source vector database, brings notable architectural and performance changes in version 2.6. Based on a sharing by technical instructor Li Jin, this article starts from the principles of vector retrieval, outlines the core features of Milvus 2.6, and explores how to build a high-performance RAG Q&A system on top of it.
Why Do We Need Vector Databases?
Traditional relational databases excel at exact matching and range queries—for example, finding a field equal to a specific value, or using LIKE for fuzzy matching. But when it comes to needs like "quickly finding a specific cat among billions of images," relational databases fall short.
A vector database's specialty is computing similarity. It converts unstructured data such as text and images into high-dimensional vectors and performs similarity retrieval over massive datasets in milliseconds via approximate nearest neighbor search. Milvus is a system built precisely for high-dimensional vector retrieval; in LLM applications, its role is analogous to MySQL in Java development—a key component for underlying data storage and retrieval.

According to Li Jin, Milvus is one of the most popular open-source vector databases globally, with tens of thousands of GitHub stars. Compared with paid solutions, the open-source version lets developers dig into the underlying principles—which is crucial for building high-performance RAG systems.
Approximate Nearest Neighbor Search (ANN) is the core algorithmic foundation of vector databases. Exact nearest neighbor search requires computing the distance between the query vector and every vector in the library one by one, with O(N) complexity, which cannot meet millisecond-level response on billion-scale data. ANN builds index structures (such as HNSW, IVF, DiskANN, etc.) to trade a tiny loss in recall for order-of-magnitude speed gains—typically reducing query time to milliseconds while maintaining recall above 95%. Different index types have trade-offs in memory usage, build time, and query accuracy, which is the origin of the "index strategy" optimization point discussed later.
High-dimensional vectors and embeddings: Unstructured data such as text and images must first be converted by embedding models into dense vectors (common dimensions 384–1536); semantically similar content is closer in vector space. Vector databases store these embedding vectors and their metadata; at retrieval time, they compute the distance between the query vector and vectors in the library (e.g., cosine similarity, inner product, or Euclidean distance) and return Top-K results.
Milvus 2.6 Architecture: Cloud-Native and Layered Design
Milvus 2.6 adopts a fully cloud-native architecture, supporting everything from single-machine deployment to cloud clusters. From the official architecture diagram, the system includes multiple node types: Stream Node, Query Node, Data Node, and a Coordinator responsible for routing requests to the appropriate nodes for different operations.

In terms of dependencies, Milvus uses etcd for metadata storage (common in the Kubernetes ecosystem), supports Kafka, Pulsar, and Warp as message middleware, and is compatible with Amazon object storage, MinIO, and cloud storage services for object storage. This modular design gives Milvus strong scalability and deployment flexibility.

The Relationship Between Open Source and Commercial Versions
It's important to clarify the relationship between Milvus and Zilliz. Zilliz is a company that offers a commercial, paid version called Zilliz Cloud based on the Milvus engine, providing online cloud services that eliminate operational hassle and run faster. This relationship is similar to that between the Linux system and commercial distributions like Red Hat/CentOS—the open-source version is free to use, while the commercial version provides enterprise-grade services.
Storage-compute separation and the role of message queues: Milvus's cloud-native architecture decouples compute from storage—Query Nodes and Data Nodes handle computation, object storage persists data, etcd stores metadata, and message middleware (Kafka/Pulsar, etc.) acts as the log broker. Write requests first enter the message queue, are consumed by Stream Nodes with ordering guaranteed, then asynchronously flushed to object storage; Query Nodes subscribe to the message queue to achieve near-real-time data visibility. This design allows each component to scale independently, and explains why message middleware is a required dependency for Milvus rather than an optional one.
Responsibilities of the Coordinator: The Coordinator is the brain of the cluster, divided into Root Coordinator (manages DDL and timestamp allocation), Query Coordinator (manages load balancing and shard allocation for Query Nodes), Data Coordinator (manages flushing and compaction for Data Nodes), and others. They do not process data themselves; they only handle metadata scheduling and request routing, thereby preventing a single point from becoming a data bottleneck.
Deployment Options and Hardware Requirements
Milvus offers multiple deployment methods. The lightest is the Python Lite version—install the pymilvus library via pip and start and connect locally. The standalone version has some hardware requirements: at least 8GB of memory and an Intel 2nd-gen Core or higher CPU are recommended, and an SSD is strongly advised—Li Jin specifically notes that Milvus basically doesn't run well on non-SSD drives.

High-Performance RAG in Practice: Key Optimization Points
The core of the course is a hands-on intelligent Q&A project based on Milvus. Li Jin emphasizes that building a RAG system is not just about storing vectors and querying them out—what really determines performance are the following optimization dimensions:
- Watermark management: Control load balancing for data writes and queries to avoid overloading a single node.
- Preload fields: Load frequently accessed fields into memory in advance to reduce query latency.
- Index strategy: Choose appropriate vector index types to balance recall and query speed.
- Tiered storage and eviction policies: Milvus 2.6 supports hot/cold data tiering; combined with eviction policies, infrequently accessed data can be moved to lower tiers to reduce memory usage.
These are exactly the "blue points" (key differentiators) often encountered in interviews and real projects, and they are what distinguishes ordinary users from architecture designers.
RAG (Retrieval-Augmented Generation) works as follows: the user's question is converted into a vector by an embedding model, the most relevant document chunks are retrieved from the vector database, these chunks are inserted into the prompt as context, and then passed to an LLM to generate an answer. Therefore, the quality of a RAG system is jointly determined by "retrieval quality" and "generation quality," and the vector database directly determines the former. Common retrieval optimization techniques include hybrid retrieval (vector similarity + keyword BM25), reranking, and metadata filtering—all of which require support from the vector database at the indexing and query layers.
Watermark mechanism: In Milvus's streaming write pipeline, watermarks mark positions in the message queue that have been persisted. Query Nodes only process data below the watermark, ensuring data visibility while avoiding reading unstable data that hasn't been flushed. Improper watermark management leads to increased query latency or delayed data visibility—which is exactly why it is listed as a key optimization dimension.
The Technical Focus of LLMs in 2026
In the course, Li Jin mentions that large models remain hot, but to establish a foothold in this field, several core problems must be solved: multi-agent collaboration, RAG optimization, model fine-tuning, and core algorithms. As the storage foundation of RAG, vector databases underpin these capabilities.
From the job market perspective, Java architects need to acquire AI knowledge, and AI engineers need to understand engineering architecture—the intersection of the two is exactly where technologies like vector databases and RAG optimization come into play. Mastering Milvus's deeper mechanisms, rather than just calling APIs, is what demonstrates differentiated value in project implementation and interviews.
Summary
Milvus 2.6 has put considerable effort into cloud-native architecture, tiered storage, and deployment flexibility. For developers who want to build enterprise-grade RAG systems, understanding the principles of vector retrieval is only the starting point; engineering details such as watermarks, preloading, indexing, and eviction policies are what determine whether the system can withstand real workloads. Rather than stopping at installation and getting started, it's better to dive deep into these mechanisms and make the vector database a truly reliable foundation for AI applications.
Related articles

Zed Editor Releases v1.29.2: Observations on a Routine Version Iteration
Zed editor releases patch v1.29.2, version bump by @JosephTLyons, commit verified via GitHub signature. This article covers the release's basic info, what a patch version means, and practical upgrade advice for users.

Anthropic IPO Questioned: Hacker News Debates Timing of Going Public
A Hacker News post titled "Just say no to Anthropic's IPO" sparks debate on the timing of Anthropic's public listing and AI capitalization, touching on valuation bubbles and mission independence.

Hands-On with the Genie 4 World Model: Directing Worlds in Real Time Like a Puppeteer
Google DeepMind's Genie 4 world model in depth: Special Guest X's hands-on demo of real-time scene direction via prompts, custom interactive apps in Project Genie, and prompts-as-seeds reproducibility.