Knowledge Graph-Enhanced RAG: Building a More Powerful Q&A System

In an era where Retrieval-Augmented Generation (RAG) has become a standard building block for LLM applications, one powerful yet often overlooked tool is quietly gaining traction — the Knowledge Graph. DeepLearning.AI, in collaboration with graph database company Neo4j, has released a short course called Knowledge Graphs for RAG that systematically explores how to combine knowledge graphs with RAG to build Q&A systems that go far beyond traditional vector retrieval. This article breaks down the core ideas and technical foundations of the course to help you quickly grasp this underrated AI tool.

What is RAG? Retrieval-Augmented Generation is a technical architecture that combines external knowledge bases with large language models, formally introduced by Meta AI in 2020. The core idea: before generating a response, the system retrieves relevant information from external data sources and feeds it as context to the model — compensating for the limitations of static training data and reducing hallucinations. RAG has become the de facto standard for enterprise AI applications, widely adopted in knowledge Q&A, customer service automation, document analysis, and more.

What Is a Knowledge Graph

A knowledge graph provides a way to store and organize data with an emphasis on relationships between things. Unlike traditional relational databases that store data in rows and columns, knowledge graphs use a graph-based structure:

• Nodes: Represent things or entities, such as people or companies;
• Edges: Represent connections or relationships between entities, such as an "employs" relationship.

A simple way to remember: when you hear "node," think "thing/entity"; when you hear "edge," think "relationship between things."

Both nodes and edges can carry additional information. For example, a "Person" node can store attributes like name and email; a "Company" node can store employee count and annual revenue; and an "employs" edge connecting a company to a person can record job title and start date.

This graph structure of nodes and relationships is highly flexible and can model the complex connections of the real world far more naturally than a relational database.

A Brief History of Knowledge Graphs The concept of the knowledge graph was formally introduced by Google in 2012 and first applied to semantic understanding in search engines. Its theoretical roots trace back to Semantic Web and Ontology research. Today, the most prominent knowledge graphs include Google Knowledge Graph, Microsoft Satori, and the open-source Wikidata. The core advantage of a knowledge graph is its ability to encode human knowledge in a machine-processable form, enabling reasoning rather than mere matching — which is the fundamental reason it can bring a qualitative leap to RAG systems.

Relationships as "First-Class Citizens"

Andres Colliger, the course instructor and Neo4j's Developer Advocate for Generative AI, emphasizes a key distinction: in a knowledge graph, relationships are a first-class part of the database itself, rather than just a shared key between two tables as in relational databases.

This design has real practical advantages — it's easier to represent and search deep relationships, queries execute faster, and relevant data can be located more efficiently. This is precisely why knowledge graphs are widely used in web search engines and e-commerce product search. When you search for a celebrity on Google or Bing, the information displayed in the side panel card is retrieved via a knowledge graph.

Neo4j and the Cypher Query Language Neo4j is the world's leading native graph database by market share, using a Property Graph Model that stores data as nodes and relationships in a native graph structure — enabling multi-hop query performance far superior to traditional relational databases. Cypher is Neo4j's declarative query language designed specifically for graph databases, with intuitive syntax: parentheses () represent nodes, square brackets [] represent relationships, and arrows --> indicate direction. For example, (Company)-[:EMPLOYS]->(Employee) expresses an employment relationship query. Cypher has been adopted by multiple graph database vendors and is becoming an industry standard for graph queries (a key reference for the GQL standard).

How Knowledge Graphs Enhance RAG Retrieval

In a basic RAG system, the typical workflow is: split documents into smaller passages or "chunks," convert them into vectors using an embedding model, then retrieve the most relevant content using methods like cosine similarity.

The Limitations of Vector Embeddings and Semantic Retrieval Embedding models are neural networks that map text into a high-dimensional vector space, where semantically similar content is positioned close together. Cosine similarity is the most commonly used similarity metric. Leading embedding models include OpenAI's text-embedding-ada-002 and the Sentence-BERT family. However, vector retrieval has a fundamental limitation: it can only capture "surface-level semantic similarity" and cannot understand structured relationships between entities. For questions involving multi-hop relationships — such as "Who is the CEO of a subsidiary of Company A?" — vector retrieval based on semantic similarity often fails to provide precise answers. This is exactly where knowledge graphs add their core value.

When we store text chunks inside a knowledge graph, entirely new retrieval possibilities emerge. In addition to traditional similarity search, you can:

1. Retrieve a relevant text chunk;
2. Then traverse the graph's relationships to find other related chunks;
3. Thereby providing the LLM with richer, more complete context.

This approach can surface connections between sources that similarity-based RAG would miss. When a knowledge graph is combined with an embedding model, you have a powerful RAG toolkit — one that leverages both the semantic similarity of vectors and the explicitly modeled relationships and metadata within the graph.

From "Find Similar" to "Find Connected"

Traditional vector retrieval is fundamentally about "finding similar" — it excels at answering "which content is semantically close to my question." Knowledge graph-enhanced retrieval adds the ability to "find connected" — answering "what else is directly or indirectly related to this piece of information."

This capability relies on Multi-hop Reasoning: answering a question requires multiple intermediate reasoning steps, chaining together multiple pieces of knowledge to arrive at the final answer. Graph Traversal is a natively supported operation in knowledge graphs — starting from a source node and progressively expanding the search along relationship edges, with BFS (breadth-first search) and DFS (depth-first search) as common strategies. This hybrid use of graph traversal and vector retrieval is the core technical differentiator of GraphRAG versus traditional RAG. Together, they allow RAG systems to perform significantly better on complex questions that require cross-document, cross-entity reasoning.

Course Practicum: Parsing SEC Financial Filings

The course is built around a real-world scenario: constructing a knowledge graph to represent the financial forms that companies are legally required to submit to the U.S. Securities and Exchange Commission (SEC). The publicly available SEC financial filing dataset is large, relationship-rich, and an ideal playground for knowledge graphs.

Why SEC Filings? The SEC requires all public companies to regularly submit standardized financial documents — including annual reports (10-K), quarterly reports (10-Q), and more — made freely available to the public through the EDGAR system. SEC filings have unique value for knowledge graph RAG research: filings contain numerous explicit entity relationships (e.g., parent-subsidiary relationships, executive roles, cross-business relationships); individual documents are information-dense and require cross-section, cross-document reasoning; and financial analysis itself demands high precision that fuzzy semantic matching alone cannot deliver. These characteristics make the SEC dataset an ideal "stress test" for validating the effectiveness of knowledge graph-enhanced RAG.

The learning path is structured progressively:

• Foundations: Introduces core knowledge graph concepts, then uses Neo4j's query language Cypher to explore and modify a movie database graph — providing an intuitive feel for how graph queries work;
• Vectorization: Combines Neo4j with text embedding models to create vector representations for text fields within the graph;
• Single-Graph RAG: Builds a knowledge graph for a set of SEC filings and uses LangChain to retrieve text from the graph, completing an end-to-end RAG workflow;
• Multi-Graph Linking: Processes a second set of SEC filings, connects the two graphs via "linked data," and uses more sophisticated graph queries to retrieve across multiple document sets.

LangChain's Role in RAG LangChain is one of the most popular LLM application development frameworks, providing standardized interfaces for connecting large language models with external tools and data sources. In a RAG architecture, LangChain plays the role of "orchestrator" — encapsulating components across the full pipeline: document loading, text splitting, vectorization, retriever construction, prompt templates, and model invocation. LangChain natively supports Neo4j integration (via components like Neo4jVector and GraphCypherQAChain), enabling knowledge graphs to serve directly as retrieval backends. It can also automatically convert natural language questions into Cypher queries, significantly lowering the barrier to building graph-powered RAG systems.

After completing these steps, learners can pose questions about the SEC dataset that involve hidden relationships across multiple companies and filings — exactly the kind of questions that pure vector retrieval struggles to handle.

A Complete Loop from Beginner to Advanced

The course follows a clear pedagogical logic: starting from basic graph concepts and Cypher syntax, progressively moving to vector embeddings and single-document-set RAG, and finally arriving at complex graph queries over multiple document sets. Every step includes hands-on practice, helping learners genuinely understand how to build a knowledge graph system from scratch.

Why Knowledge Graphs Deserve a Place in Your AI Stack

Andrew Ng describes knowledge graphs in the course as "one of the most powerful, yet relatively underappreciated tools" in the AI field. Amid the LLM boom, vector databases have captured enormous attention, while knowledge graphs — a technology capable of explicitly representing structured relationships — have remained on the periphery.

But as RAG applications push into more complex and specialized domains, the limitations of pure similarity retrieval are becoming apparent — it struggles with questions requiring multi-hop reasoning and understanding of relationships between entities. Knowledge graphs fill exactly this gap. According to the course, Neo4j's Zachary Blumenfeld, along with DeepLearning.AI's Tommy Nelson, Jeff Ludwig, and other team members, all contributed to the course's development — a sign of how seriously the industry is taking this direction.

For developers looking to make their RAG systems "smarter about relationships in data," mastering knowledge graphs can not only improve retrieval accuracy and completeness, but may also help you discover important connections in your data that you never noticed before — and that is the unique value of knowledge graph-enhanced RAG.

Key Takeaways