RAG Source Attribution Broken? 5 Battle-Tested Chunk-Level Provenance Lessons

When building RAG (Retrieval-Augmented Generation) systems in production, many engineers find that their biggest headache isn't retrieval quality — it's a more insidious and far more damaging problem: source attribution degrades as data flows through the entire pipeline. A developer who has spent years building production-grade RAG systems with LangChain + Elasticsearch recently shared the complete thinking behind his open-source project auditrag, which was built specifically to address this problem. These lessons apply to virtually any tech stack.

What are RAG and source attribution? RAG is an architectural paradigm that combines large language models with external knowledge bases — when generating an answer, the model first retrieves relevant text chunks from a vector database, then injects those chunks as context into the prompt. Source attribution refers to annotating, within the final answer, which specific original document fragment each claim comes from. In high-stakes domains like law and medicine, this isn't just a UX concern — it's a fundamental requirement for compliance and auditing.

The Core Problem: Why RAG Citations Are So Often Untrustworthy

By the time an answer reaches the user, the so-called "citation" has typically degraded into a vague document-level link that gives users no way to verify whether a specific claim is actually backed by the source text.

The developer puts it plainly: most provenance loss happens when IDs are regenerated or re-keyed somewhere in the middle of the pipeline. Data passes through multiple stages — document chunking, vectorization, retrieval, and generation — and at each step, the original "evidence anchor" can silently snap. By the time the answer reaches the user, you can no longer answer the most basic question: "Which exact passage does this sentence come from?"

For audit-adjacent scenarios in finance, law, and healthcare, this unverifiability is simply unacceptable. This pain point is what drove him to bypass the frameworks and redesign a provenance mechanism from the ground up — one that can be strictly enforced end-to-end.

5 Transferable Lessons for Chunk-Level Provenance

1. Mint Deterministic Chunk IDs Once, at Ingestion Time

The first principle: during the ingestion stage, generate a deterministic, immutable ID for each text chunk — for example, using a composite format like {doc_hash}:{page}:{chunk_index}.

The key words here are "deterministic" and "immutable." Once generated, this ID must pass through every stage of the pipeline unchanged. The root cause of most provenance loss is IDs being regenerated mid-pipeline. Hold this invariant, and you have a reliable tracking baseline you can actually trust.

Why deterministic IDs? The core idea behind deterministic IDs comes from content-addressable storage. By computing a hash (e.g., SHA-256) of the document content combined with page number and chunk index, you generate a unique identifier that is bound to the content — the same content always produces the same ID, no matter how many times it's processed. This is fundamentally different from randomly generated IDs like UUIDs: UUIDs produce a new value every run, causing provenance chains to silently break across batches — often without the developer ever noticing.

2. Never Let the LLM See Real IDs

This is the most elegant design in the entire approach. Don't give the model the real chunk IDs. Instead, give it small integer labels like [1], [2], [3], and maintain a "label → real ID" mapping table scoped to each request.

This design has a critical side effect: if you only sent 5 chunks but the model cites [7], you've immediately caught a hallucinated citation — rather than silently resolving it to some unrelated content.

The engineering logic behind detecting hallucinated citations LLM "hallucinations" are especially dangerous in citation scenarios — the model may confidently cite a source that simply doesn't exist. The integer label mapping mechanism is essentially an engineering implementation of the "closed-world assumption": the system predefines the boundary of all valid citations, and any out-of-bounds citation is treated as an anomaly rather than valid input. This is analogous to a database foreign key constraint — it transforms an otherwise invisible semantic error into a structural error that can be detected programmatically and immediately, turning "the model invented a citation" from a silent, hidden failure into an observable, explicit anomaly.

3. Treat the Vector Store as a Pure Index

A vector store is fundamentally just an index — don't treat it as your source of truth. The actual chunk text should live in a "boring but reliable" store — the author chose SQLite.

The logic behind this principle is decoupling: your validation and reporting pipeline should never depend on the internal implementation of the vector store. The vector store handles similarity retrieval; fact-checking and citation reporting read from an independent source of truth. This means that even if you swap out your vector store, your provenance and auditing capabilities remain intact.

The architectural thinking behind separating index from data Vector databases (e.g., Elasticsearch, Pinecone, Weaviate) are optimized for efficient approximate nearest neighbor (ANN) retrieval — not transactional storage. Storing raw text chunks in a relational database like SQLite reflects the classic architectural principle of "separating index from data": the vector store holds embedding vectors and minimal metadata, while the authoritative copy of the original text lives in independent persistent storage, linked together by immutable IDs. Even if the vector index is corrupted, migrated, or replaced entirely, auditing and provenance capabilities remain fully intact.

4. Add a Faithfulness Verification Pass

After generating an answer, the author adds a faithfulness pass: each claim made by the model is compared against the verbatim source text of its cited chunk, and given one of four verdicts:

• supported
• partial
• unsupported
• uncited

He openly acknowledges the reliability concerns with "using an LLM to judge an LLM," but takes a pragmatic view: flagging a suspicious claim is always better than doing nothing. This is defensive engineering thinking — a noisy check is better than letting errors silently flow to users.

Technical background on faithfulness evaluation Faithfulness is one of the core metrics in RAG evaluation frameworks (such as RAGAS and TruLens), measuring whether each claim in the generated answer can be supported by the retrieved context. Using an LLM to judge another LLM is called "LLM-as-judge." Its limitations include the judge model itself being prone to errors and systematic biases (e.g., a tendency to score confidently worded answers higher). However, research shows that at scale, LLM judges correlate well with human annotations — making them a reasonable engineering tradeoff between cost and precision, especially as a real-time defense layer in production systems.

5. Respect Page Boundaries — Never Chunk Across Pages

The final recommendation: never let a text chunk span a page boundary.

The benefit is that every claim can be mapped to a precise page number, making citations verifiable and locatable. The author acknowledges this slightly hurts retrieval quality (chunking can no longer be fully semantics-optimal), but for audit-related applications, "it's 100% worth it." This is a deliberate tradeoff between retrieval recall and verifiability.

Why Go Framework-Free

One implementation detail worth noting: the author deliberately chose a framework-free approach — auditrag is not built directly on top of LangChain.

His reasoning is straightforward: only by stepping outside the framework can you strictly enforce ID invariants end-to-end. This reflects a real tension in current RAG engineering practice — mainstream frameworks like LangChain and LlamaIndex, in pursuit of generality and ease of use, introduce heavy abstraction and transformation between document metadata and pipeline stages. For example, LangChain's Document object goes through multiple rounds of serialization and deserialization during chunking, retrieval, and passing — each one a potential point of silent metadata loss. These abstraction layers are great for rapid prototyping, but become a systemic obstacle when you need strict chunk-level provenance.

He closed his post with an open question to the community: has anyone successfully implemented strict chunk-level provenance inside LangChain? He couldn't find a clean solution himself, and suspects it might be achievable through custom Document metadata combined with callbacks. This remains an engineering question worth exploring in depth.

Summary: Building RAG Systems That Can Actually Be Trusted

The value of these lessons lies not in any single trick in isolation, but in the provenance invariant philosophy they collectively form: IDs are minted once and never changed; the model is isolated from real IDs; the data store is decoupled from the index; generation is followed by mandatory verification; chunking respects physical boundaries.

For any team building production-grade RAG systems — especially those in audit-sensitive domains — these principles are highly worth adopting. Retrieval quality matters, of course, but a system only earns the right to be trusted in serious applications when every claim the AI makes can be traced back and verified.