Building a GraphRAG AI Agent from Scratch: A Complete Practical Guide

Why GraphRAG?

Traditional RAG (Retrieval-Augmented Generation) works reasonably well for simple Q&A, but struggles with complex questions that require reasoning across multiple documents or passages. The original video uses a vivid analogy: standard RAG is like a "keyword highlighter" — it scans documents for matching terms and pulls out a few paragraphs. If the answer requires connecting clues from page 2 and page 20, it often gets confused or misses those connections entirely.

GraphRAG is more like a "mind map." Before answering, it reads through the documents and builds a connected network of real-world entities (people, places, chemical substances, animals, etc.) and their relationships (causes, eats, flows into, etc.). When faced with a complex question, GraphRAG follows the edges in the graph to instantly link distant facts scattered across different documents.

This is precisely the core scenario this tutorial addresses — questions like "How do ship traffic and busy shipping lanes ultimately damage the shells of marine organisms?" require multi-hop reasoning that standard vector search simply can't handle with a proper logical chain.

System Architecture: Data Flow from Documents to Answers

The system uses a Dual Indexing design, with data flowing from Python scripts and MCP tools all the way to the frontend interface. Core components include:

• Vector Store: Documents are split into overlapping text chunks (600 characters, 100-character overlap), embedded using NOMIC embed text, and stored in ChromaDB.
• Knowledge Graph: A Gemma model extracts "entity–relation–entity" triples, which are built into a NetworkX DiGraph and persisted as graphstore.json.
• Service Layer: OpenWebUI sends request payloads to a FastAPI service running on port 8000.

This dual-track "vectors + graph" approach is what sets GraphRAG apart from traditional solutions. The vector index handles semantic similarity matching, while the graph index handles structured relational reasoning — two complementary capabilities.

MCP (Model Context Protocol) is an open protocol introduced by Anthropic in 2024 to standardize how AI models interact with external tools and data sources. Think of it as the "USB port" of the AI world — tool developers expose interfaces according to the protocol, and any MCP-compatible agent framework can plug in and call those tools without platform-specific adaptation. FastMCP is its Python implementation library, which greatly simplifies registering ordinary Python functions as MCP tools. In this system, three retrieval capabilities (vector search, entity lookup, and path synthesis) are wrapped via FastMCP so the orchestrator can dispatch them as structured tools.

NetworkX is the most widely used graph analysis library in the Python ecosystem. It supports directed graphs (DiGraph), undirected graphs, multigraphs, and more, with built-in algorithms for shortest paths, connectivity analysis, centrality computation, and many others. This system uses a directed graph to store "entity → relation → entity" triples — directionality ensures that "A causes B" and "B causes A" are treated as distinct edges. In path synthesis scenarios, the system temporarily treats the graph as undirected to discover bidirectional implicit connections.

Document Ingestion and Indexing Script

The first core script processes local documents into the dual indexing system. Its workflow is straightforward:

1. Define directory paths and model names (Gemma for LLM tasks, NOMIC embed text for embeddings), and initialize a persistent ChromaDB vector collection at /chroma_db.
2. Iterate over the /documents directory, reading all .txt and .md files as document objects.
3. Send text chunks to Gemma via Ollama at zero temperature to extract structured JSON triple lists (source, target, relation).

The temperature parameter is worth noting: temperature controls the randomness and creativity of LLM output. Lower temperatures produce more predictable, precise, and factually grounded responses — for entity extraction tasks that demand accuracy, zero temperature is the right choice.

For each text chunk, the script simultaneously does three things: generates and upserts vector embeddings into ChromaDB; extracts triples and adds nodes and edges to the NetworkX DiGraph; and finally exports the complete graph structure to graphstore.json.

Three Retrieval Tools: The Role of MCP

The second script uses FastMCP to wrap three retrieval capabilities into tools that downstream AI agents can call:

Vector Hybrid Search

Performs similarity search in ChromaDB using pre-computed query embedding vectors, returning top-K matching document passages along with file metadata. Best suited for unstructured semantic queries.

Entity Lookup

Locates target nodes in the NetworkX graph using case-insensitive substring matching, collects all direct incoming and outgoing relationships for that node, and summarizes its neighborhood within the knowledge graph.

Path Synthesis

This is where GraphRAG's power truly shines. It treats the graph as undirected, computes the shortest path between entity A and entity B, finds direct or multi-hop connections, and generates an explanation of how the two entities are related.

Multi-hop Reasoning refers to answering a question that requires two or more intermediate reasoning steps, where each step's conclusion becomes the input for the next. For example, to answer "How does ship traffic damage marine organism shells?", the system first needs to know that "ship traffic produces noise and pollutants," then that "pollutants cause ocean acidification," and finally that "acidic water dissolves the calcium carbonate in organism shells." This reasoning chain spans multiple document fragments. A single vector similarity search can only find the passage most similar to the question's phrasing — it cannot automatically chain together intermediate nodes. Knowledge graph shortest-path algorithms are naturally suited to this task: they can traverse intermediate nodes in the graph and connect scattered facts into a complete causal chain.

Intelligent Routing: Intent Classification and Multi-Model Collaboration

The third script is the agent's brain — the GraphRAG helpdesk orchestrator. It imports the MCP tools created earlier and implements a dual-model collaboration mechanism:

• Fast model (Gemma 2B): Handles intent routing. The intent router analyzes the user's request and classifies it into one of four execution paths — vector_search, entity_lookup, path_synthesis, or general.
• Reasoning model (Gemma 4): Handles complex tasks and final answer synthesis, with automatic fallback error handling.

Different intents follow different processing branches: semantic queries go to vector hybrid search; entity-type questions parse out the target entity name and call the graph neighborhood lookup; relationship-type questions extract an entity pair and call path synthesis. This "classify first, then handle specifically" design ensures each type of question is processed by the most appropriate tool.

Using two models of different scales for division of labor (2B parameters for routing, 4B or larger for reasoning) is a classic Cascading Inference strategy. The core logic is: "use the minimal compute cost for classification, then concentrate resources on the steps that actually require reasoning." Intent classification is essentially a four-class problem with fixed semantic patterns — a small model handles it easily with lower latency. Synthesizing multi-source retrieval results into a logically coherent answer, on the other hand, demands stronger contextual understanding and text generation capabilities. This design also brings observability benefits in engineering practice: intent classification results can be logged and monitored, so when the system misclassifies a query type, developers can quickly pinpoint whether the issue lies in the routing layer or the execution layer.

FastAPI Wrapper and OpenWebUI Integration

The final script wraps the GraphRAG Agent into a FastAPI service compatible with the OpenAI API. Its responsibilities include:

• Configuring Pydantic request models and instantiating the orchestrator (Gemma 4 for complex tasks, Gemma 2B for fast routing).
• Parsing OpenAI-compatible message arrays, extracting user text prompts, and decoding base64-encoded image URLs into temporary local files to support vision processing.
• Exposing OpenAI-style endpoints that list the graphrag_router_agent model, allowing frontends like OpenWebUI to discover and select this agent.
• Running the application with Uvicorn on 0.0.0.0:8000 to handle HTTP requests from local or Docker clients.

On the frontend side, OpenWebUI runs in a Docker container on port 3000. Simply configure it to use the GraphRAG Agent and you're ready to chat.

Real-World Performance: The Power of Multi-Hop Reasoning

The tutorial validates the system with several interesting questions. For a question like "How do ship traffic and busy shipping lanes damage the shells of marine organisms?", the fast model first extracts key entities (ship traffic, shipping lanes, marine organism shells), the path expert queries the graph and traverses intermediate nodes, and the reasoning model synthesizes the retrieved graph connections into a complete logical causal chain — categorized into acute trauma, chronic degradation, and systemic weakening.

Another question — "Is there a hidden connection between squid in the Pacific and the Gulf Stream in the Atlantic?" — similarly demonstrates cross-document analysis capabilities. These answers are difficult for standard RAG to produce, because they require connecting facts scattered across different documents.

Tech Stack Summary

The technology choices in this solution are quite representative for locally deployed AI Agent implementations:

• Embedding Model: NOMIC embed text
• LLM: Gemma (2B for fast routing, 4 for reasoning and synthesis)
• Vector Store: ChromaDB
• Knowledge Graph: NetworkX DiGraph
• Tool Protocol: FastMCP
• Service Framework: FastAPI + Uvicorn
• Frontend: OpenWebUI

The overall architecture clearly demonstrates how to combine vector retrieval with a knowledge graph to build an AI Agent with genuine reasoning capabilities. For developers looking to enable complex Q&A over private documents, this is a complete and well-referenced implementation path.