Structuring Data for AI Agents: Five Approaches and a Selection Guide

Why AI Agents Keep "Confidently Getting It Wrong"

Ask five vendors how to prepare data for AI Agents and you will get five different answers — and each answer happens to be exactly the product that vendor is selling. Graph database companies say knowledge graphs. Vector database companies say RAG. Semantic layer companies say semantic models. They are all correct, each describing the one tool in their own toolbox.

But the honest answer is: there is no single optimal approach to structuring data for AI Agents. Five approaches are currently used in production for real — semantic models, ontologies and knowledge graphs, retrieval-augmented generation (RAG), plain Markdown files with grep search, and open knowledge formats. Each one addresses a different failure mode. Pick the wrong one and the Agent won't loudly error out; instead it will quietly produce a fluent, confident, and wrong answer. The problem is never the model itself — it is the data fed into it.

What "Structuring Data for AI Agents" Actually Means

Structuring data means organizing information so an Agent can reliably find the correct fact, definition, or document. The opposite is guessing from raw, unlabeled data and then being wrong in a way that sounds right. Enterprise AI Agent failures rarely happen because the underlying model is too weak — they happen because nobody told the Agent what the definition of an "active customer" is, or where the answer to a given question lives.

This is a data problem, not a prompt problem. A model handed a clean, governed metric definition will use it correctly. A model handed a pile of raw tables without context will invent a plausible-looking calculation that only gets exposed when someone cross-checks the numbers in a meeting.

The Five Approaches to Structuring Data

1. Semantic Models

A semantic model (or semantic layer) defines business metrics and entities once: "monthly recurring revenue," "active customer," "churn rate," along with the exact tables, joins, and filters behind each metric. Agents query metrics by name instead of writing SQL from scratch against raw tables.

This is the highest-leverage fix for one common failure mode: Agents guessing at business definitions and arriving at a number that sounds right but isn't. In dbt Labs' 2026 benchmark, Agents querying a semantic model hit 98.2%–100% accuracy on a set of business questions, while the same models writing raw text-to-SQL against a full schema achieved 84.1%–90.0%. That benchmark also compared today's models against those doing raw text-to-SQL in 2023, finding accuracy nearly doubled over 2.5 years, from 32.7% to 64.5%. That is real model progress. But even the best 2026 models still trail the semantic-model numbers. The more direct takeaway: the gap between a wrong query and a right one comes from what the Agent is grounded in, not how strong the underlying model is.

Semantic models fit recurring business metrics — repeatedly queried across multiple tools and multiple Agents, where "everyone getting the same number" matters. They don't fit one-off questions about an individual document, nor scenarios that connect facts across entities.

Semantic models also have a value the other four approaches can't automatically match: define a metric once, and any AI Agent can query it via MCP (Model Context Protocol, the open standard connecting Agents to data sources), without every team building custom retrieval or graph layers to answer the same question. This turns semantic models from a one-off project into shared infrastructure. Both dbt and Cube offer semantic layers — which tool you choose matters less than taking metric definitions seriously.

MCP (Model Context Protocol) is a protocol open-sourced by Anthropic in late 2024, intended to provide a unified connection standard between AI applications and external data sources and tools. Before MCP, connecting each data source (database, SaaS API, file system) typically required its own dedicated adapter, leaving Agents and data in an N×M fragmented relationship. MCP abstracts this into a client–server model: a data source exposes an MCP server, and any MCP-compatible Agent can discover and call its resources through the same interface. For a semantic layer, this means defining a metric in one place, and every Agent (regardless of which model or framework it runs on) can query it by metric name without rebuilding retrieval or graph layers.

The point of intersection between MCP and semantic models is this: the semantic layer itself is a "governed catalog of metrics and dimensions," a natural fit to be exposed as an MCP server. Agents no longer generate SQL; instead they issue a semantic query, which the semantic layer translates into controlled SQL and returns a certified number. This also explains why the article stresses that MCP turns the semantic model from a one-off project into shared infrastructure — the protocol lowers the integration cost of cross-team, cross-Agent reuse.

2. Ontologies and Knowledge Graphs

Knowledge graphs store entities (people, accounts, products, contracts) and the explicit relationships between them, rather than flat tables or text chunks. An ontology is the schema defining what those entities and relationships mean, keeping the graph coherent as it grows. Combined, they let Agents traverse connections that keyword or embedding matching entirely misses.

This is where the name GraphRAG comes from: retrieval that queries a knowledge graph, plus generative summarization over clusters of related entities, rather than a flat vector index. Microsoft's 2024 research found GraphRAG won 72% of head-to-head comparisons against ordinary vector RAG on comprehensiveness, and 62% on diversity. Its top-level summaries needed over 97% fewer tokens than directly summarizing the source text. A separate 2025 evaluation of RAG and GraphRAG systems found that adding selection and integration strategies on top of graph retrieval improved multi-hop reasoning benchmarks by 1.1% to 6.4% over the best baseline. That is a real but more modest gain than GraphRAG's headline numbers, and a reminder that retrieval technique itself matters as much as the graph.

Knowledge graphs cost more to build and maintain than the other four approaches. Someone has to model entities and relationships, and keep the ontology accurate as the business changes. That cost is worth paying when questions genuinely require multi-hop reasoning across connected entities. If the question is just "look up a fact," it's over-engineering.

GraphRAG is a retrieval-augmented generation variant proposed by Microsoft Research in 2024. Ordinary vector RAG chunks documents, embeds them, and retrieves the most relevant few by similarity — it is naturally good at "local" questions but struggles to answer "what is the corpus as a whole about?" or questions that require connecting multiple entities. GraphRAG's approach is to first have a large model extract entities and relationships from the corpus, build a knowledge graph, run community detection on the graph, and pre-generate hierarchical summaries for each community. At query time it can do local entity-neighborhood retrieval or answer global questions using community summaries.

The two evaluations mentioned in the article should be read together: Microsoft's 2024 research showing a 72% comprehensiveness win rate and 97% token savings is GraphRAG relative to naive vector RAG, and comes from the proposing party itself; the 2025 independent evaluation decomposed the graph retrieval pipeline and found that what actually delivers stable gains is the "selection and integration" strategy on top of retrieval, improving multi-hop benchmarks by only 1.1%–6.4%. The two numbers don't contradict — they measure "whole-system comparison" and "single-component ablation" respectively, and the latter is closer to the gain that can be reliably reproduced in real engineering.

3. RAG (Retrieval-Augmented Generation)

RAG retrieves the most relevant text chunks (usually via embedding similarity in a vector database) and hands them to the model as context before it answers. This is the fastest approach to build: chunk, embed, index, and you can have it running in an afternoon.

RAG is the correct default for fact lookup across large, loosely structured document sets: contracts, tickets, product docs, meeting notes. It is the wrong fit when the answer requires connecting several facts across multiple documents (that's the knowledge graph territory), and when the "fact" is a governed metric that should have exactly one certified answer everywhere (that's the semantic model territory).

There is also a subtler trap: more retrieved context does not automatically mean better. Chroma's 2025 research tested 18 frontier models (including GPT-4.1, Claude Opus 4, and Gemini 2.5) and found accuracy does not degrade uniformly as input length grows. Even a single distractor paragraph measurably hurts performance relative to a cleaner baseline. Models sometimes perform worse on logically well-structured long contexts than on the same information shuffled. This finding has been widely validated enough to serve as a rule of thumb: retrieval precision always beats quantity.

Some teams have achieved real gains by combining approaches: retrieve with a vector index, then check or expand the answer using graph relationships. AWS's 2024 case study with Lettria showed that layering graph-based retrieval on top of vector RAG improved answer precision by up to 35% across finance, healthcare, aviation, and legal use cases.

4. Plain Markdown + grep

The most unglamorous option, and the one no infrastructure vendor will ever recommend — because nobody sells it: put your data in well-organized plain Markdown files and let the Agent search it with grep or another full-text tool. No embeddings, no vector database, no graph to maintain.

For small, well-organized internal corpora (a few hundred files, not millions), this works far better than it sounds. Coding Agents already use this heavily. A well-structured repo of Markdown documents, read directly by an Agent with a file search tool, is often faster and easier to understand than standing up retrieval infrastructure for a corpus that fits on a laptop. This is the intuition behind emerging patterns like llms.txt and skill-file conventions: greppable plain text organized by someone who knows the content beats a half-tuned vector index over the same material.

There is an easily underestimated maintenance advantage here: Markdown files live in version control like everything else. Every change is a human-readable, reviewable, rollback-able diff. No embedding pipeline to re-run, no index to rebuild, no vector database to keep in sync with the source of truth.

This approach fails where you'd expect: once the corpus grows past a certain size, or the wording gap between a query and the correct document grows wide enough, keyword matching starts missing what semantic or graph search would catch. That's the signal to upgrade to RAG or a knowledge graph — not a reason to skip Markdown+grep as a reasonable starting point.

llms.txt is a convention proposed in 2024: place a Markdown file at a site's root directory that uses concise headings, links, and summaries to tell large language models what content the site has, what's important, and in what form it's presented. Its positioning is similar to robots.txt for crawlers, but its audience is AI assistants and Agents. skill-file is a similar pattern in Agent frameworks: a human-readable plain-text spec describing "what this Agent can do, in what steps, and what inputs it needs," so the Agent reads it directly instead of relying on implicit fine-tuning.

Both share the same intuition: for a modest-sized corpus manually organized by people who know the content, plain text searchable by full-text is more reliable and more reviewable than a half-tuned vector index. It de-emphasizes the "build infrastructure first, then answer" mindset — the file itself is under version control, changing a line is a rollback-able diff, and there's no embedding to rebuild. This pattern holds for small corpora, but as the corpus grows or the semantic gap between query and document wording widens, keyword matching starts to miss, and that's when it's time to introduce RAG or a graph.

5. Open Knowledge Formats

Open knowledge formats like schema.org, JSON-LD, and RDF use shared public vocabularies to structure data, rather than a format only your own system can interpret. The first four approaches structure data for your own Agent; open formats only make sense when data needs to be read by Agents you don't control: a customer's AI assistant looking up your product pricing, a search engine's AI Overview summarizing your docs, or a partner's Agent extracting structured facts from your public site.

This is a different task from the other four, and it is easy to conflate. Semantic models, knowledge graphs, or RAG indexes are internal infrastructure serving your own Agents. Open formats are a publishing choice, closer to what you're already doing when marking up web pages for search engines — except the audience reading those markers increasingly includes AI Agents, not just crawlers. If your data only needs to serve your own team's Agents, this is the one item on the list you can reasonably skip.

schema.org, JSON-LD, and RDF are three open knowledge representation approaches often mentioned together, but they operate at different layers. schema.org is a shared vocabulary, agreeing on common names for concepts like "Product," "Offer," and "Organization" and their properties, so different sites can understand each other's descriptions of the same thing. JSON-LD is a serialization format that encodes RDF graph data into JSON, so it can be embedded directly in a web page's <script type="application/ld+json">, making it friendly to both humans and machines. RDF is the lower-level triple model (subject–predicate–object), the foundational data model of the Semantic Web.

The key difference between these three and the previous four approaches is not technical but about audience: semantic models, graphs, and RAG indexes are internal infrastructure serving your own Agents, while open formats are a publishing act aimed at external consumers. It's closer to structured markup for search engines, except that now the readers of those markers include not only traditional crawlers but also search engines' AI Overviews, customers' AI assistants, and partners' Agents. Precisely because the audience is outside your control, a shared vocabulary matters more than a custom schema — otherwise external Agents cannot interpret your data.

Five Approaches at a Glance

Approach	Best For	Build Cost	Maintenance	Biggest Weakness
Semantic Models	Governed business metrics, recurring questions	Medium: one-time metric modeling	Medium: update as metrics evolve	No help for unstructured documents
Ontology/Knowledge Graph	Multi-hop reasoning across connected entities	High: model entities and relationships	High: keep the ontology accurate	Expensive even for simple lookups
RAG (vector search)	Fact lookup across large, loose corpora	Low: chunk, embed, index	Low to medium: re-index as content changes	More context isn't automatically more accurate
Plain Markdown + grep	Small, well-organized internal corpora	Very low: organize the files well	Low: no infrastructure to run	Breaks down as the corpus grows or ambiguity increases
Open formats	Data read by Agents you don't control	Low to medium: mark up public content	Low: mostly set and forget	Doesn't structure data for your own internal Agents

Which One Should You Use?

Match the structure to the question, not to the vendor pitch. In practice, that means asking what kind of question the Agent needs to answer:

• "What is our [metric]?" A recurring, governed business number. Use a semantic model.
• "How does X connect to Y, and what does that chain mean?" Multi-hop reasoning across entities. Use an ontology or knowledge graph, optionally with GraphRAG-style retrieval.
• "What does this document say about ...?" Fact lookup across a large corpus. Use RAG.
• "What is our process?" A corpus small enough that you roughly know where the answer is. Plain Markdown + grep is often faster to build and just as reliable.
• "Can an Agent I don't control read this?" A partner's, customer's, or search engine's AI. Use open formats.

Most production systems combine two or three approaches rather than picking just one: a semantic model for metrics, RAG for contracts and tickets, greppable internal docs for team runbooks, all read by the same Agent. The mistake isn't picking a single approach — it's assuming one approach has to cover every question.

How to Land This in Practice

Take a revenue operations team building a renewal assistant Agent. In a single conversation, this Agent might face five different kinds of questions, each requiring a different structure behind it.

"What is this account's net revenue retention?" — A governed metric with exactly one correct calculation. The Agent should query a semantic model, not write its own SQL against raw subscription tables and hope it matches finance's number.

"Who reports to whom at Acme Corp, and who among them is involved in the renewal decision?" — A multi-hop question across people, roles, and accounts. This is a knowledge graph question. Flat tables or text search can't trace a reporting chain.

"What does the contract say about the renewal notice period?" — A fact buried in one of thousands of contracts. RAG retrieves the relevant clause without anyone reading every file.

"Do we have an internal playbook for at-risk renewals?" — Points to a few dozen well-organized runbook docs the team wrote and knows intimately. Plain Markdown + grep finds the right one in milliseconds with zero infrastructure to maintain.

"Is our public pricing page understandable to a customer's own AI assistant?" — A question for an Agent you don't control. That's the open formats scenario.

No single approach answers all five. That is the real argument for treating "how to structure data for AI Agents" as a portfolio decision rather than a one-time architecture choice.

The migration path tends to run one direction: teams start with Markdown+grep because it's the fastest to get working. When the corpus outgrows keyword search, they migrate part of it to RAG. Only when they hit a specific kind of question — usually a multi-hop question RAG keeps getting wrong — do they add a knowledge graph. Few teams go to a knowledge graph on day one, and that's a reasonable default: build the structure the current problem needs, and add more only when a real problem exposes the gap.

Context Engineering vs. Prompt Engineering

All of the above is the practice now known as context engineering: deciding what information reaches the model's context window, in what form, and how much of it. This is different from prompt engineering, which is about how you phrase instructions in a single message. The distinction matters because a perfectly worded prompt cannot fix a model fed the wrong facts, or too many of them.

Regardless of how you structure data — semantic model, knowledge graph, RAG, or a Markdown file — the same discipline applies: give the Agent exactly the facts needed to answer the question, not everything you have.

Conclusion

Five real approaches, five different jobs. Semantic models for governed metrics, knowledge graphs for multi-hop reasoning, RAG for fact lookup across large corpora, plain Markdown+grep for small corpora, open formats for Agents you don't control. Vendors selling each approach will tell you theirs is the answer. It usually isn't — at least not on its own. Start from the questions your Agent needs to answer, and the right structure emerges naturally, rather than starting from which infrastructure is easiest to sell.

One-line Summary

There is no single optimal way to structure data for AI Agents — match five structures to the question and combine them.

Paragraph Summary

The article argues that enterprise AI Agent failures rarely stem from weak models, but from unstructured data, causing Agents to quietly and confidently answer wrong. The five approaches actually used in production each address a different failure mode: semantic models solve guessed governed metrics, knowledge graphs solve multi-hop reasoning across entities, RAG handles fact lookup across large loose corpora, plain Markdown+grep handles fast retrieval over small internal corpora, and open knowledge formats publish data for Agents you don't control. The article stresses that the choice should be based on what questions the Agent must answer, not on vendor pitches; most production systems combine two or three approaches, and the migration path typically starts with Markdown+grep, moves to RAG as the corpus grows, and adds a knowledge graph only when multi-hop questions keep getting answered wrong.