RAG in Production: Hard-Won Lessons on Document Version Control and Scanned Document Processing

Two Underrated Production Challenges in RAG

Discussions around retrieval-augmented generation (RAG) systems tend to focus on vector search algorithms, embedding model selection, or prompt engineering. Yet a developer stress-testing a "provenance-heavy RAG knowledge system" in a real-world business setting recently surfaced two problems that genuinely keep engineers up at night — yet are rarely discussed in a systematic way:

• Documents that change over time: Policies, specifications, manuals, pricing pages, contracts — these are not static. They get revised, renamed, split, and merged on an ongoing basis.
• Scanned documents and complex layouts: OCR, tables, forms, multi-column layouts, handwritten annotations, low-quality scans, and more.

What these two categories have in common is that they barely exist in idealized architecture diagrams, yet they repeatedly cause retrieval failures in real-world document corpora. Drawing on practitioner experience and lessons from the community, this article breaks down the nature of these two challenges and outlines practical strategies for addressing them.

Document Version Control: Preserving "Identity" Through Evolution

Treating Sections as Stable Lineage Units

One key practice is: treat document sections as stable lineage units, versioning each revision at the section level rather than treating the entire document as an indivisible black box.

The logic is straightforward. In a contract or product manual, the majority of content stays stable across iterations — what actually changes is usually a handful of clauses or parameters. If you re-chunk and re-embed the entire document on every update, you not only waste compute, but more critically, you lose the continuous identity of the content — the system becomes unable to answer provenance questions like "what did this clause say in the previous version?"

The concept of "lineage" is borrowed from the data engineering world, where it refers to tracing the complete flow of data from source to downstream consumers. In a RAG context, document lineage specifically refers to the metadata chain that can answer questions like: "Which document did this chunk come from? Which section? Which version? When was it last modified?" A common approach for managing section-level lineage is to assign each chunk a stable logical section ID that remains constant across version iterations, while fields like version number, revision timestamp, and content hash are written fresh on each update. This way, even when section content changes, the system can still link different versions of a chunk via the logical ID — enabling provenance queries like "compare section 2.1 between version 3 and version 5" without treating every update as entirely new content.

Handling Renames, Moves, Splits, and Merges

The truly tricky edge cases are section renames, moves, splits, and merges. When one section is split into two, or two sections are merged into one, how should the system determine correspondence between old and new content?

A useful strategy here is: don't let semantic similarity automatically resolve these ambiguous cases — route them to a human review workflow instead. This is an important engineering judgment call. Semantic similarity is highly error-prone for version alignment. A clause that is worded similarly but carries a different legal meaning could be incorrectly flagged by the algorithm as "same content," corrupting the entire provenance chain. In high-stakes domains (contracts, compliance, pricing), passing uncertainty to a human rather than a probabilistic model is often the more responsible choice.

Preventing Stale Embeddings from Silently Winning Retrieval

The most insidious failure mode is what practitioners call stale embeddings silently winning retrieval.

When a document is updated, if the old version's vectors aren't promptly cleaned up or deprioritized, they may be recalled first simply because they're semantically "closer" to the query — causing the system to return outdated policies or prices. This type of failure doesn't throw an error; it just quietly returns the wrong answer. Common countermeasures include:

• Attaching a version timestamp and validity flag to each chunk and applying metadata filtering at retrieval time;
• Introducing recency-aware reranking as a post-retrieval step;
• Hard-isolating deprecated versions rather than relying solely on similarity-score competition.

Recency-aware reranking is a post-processing step added after standard vector retrieval. It incorporates document timestamps or version numbers as an additional signal, combining them with semantic similarity scores via weighted aggregation to bias rankings toward more recent content. A common implementation computes a composite score for each retrieved candidate chunk: final_score = α × semantic_score + (1-α) × recency_score, where recency_score decays the further a document's update date is from the present. The importance of this mechanism lies in the fact that pure semantic models have no inherent sense of time — a 2020 policy document and its 2024 update may be highly similar in vector space, making them indistinguishable by similarity alone. Note that the value of α requires careful tuning based on business context: in compliance or contract scenarios, recency should carry more weight; in historical archive retrieval, the opposite strategy may be warranted.

Scanned Documents and Complex Layouts: Where OCR Breaks Down

Layout-Aware Extraction Beats "Flatten Everything to Text"

Practical experience consistently shows that layout-aware extraction significantly outperforms naively flattening all content into plain text when processing PDFs.

This is especially critical for documents with tables or multi-column layouts. When a two-column page is read line by line, text from the left and right columns gets interleaved, completely destroying the semantics. When a table is flattened into a string of text, row-column relationships are lost entirely. Preserving layout structure means the system can understand "which row and column does this number belong to" — which is crucial for downstream retrieval accuracy.

Current mainstream layout-aware PDF extraction approaches fall into two broad categories. The first is rule-based tooling (e.g., pdfplumber, PDFMiner), which reconstructs layout structure by analyzing coordinate metadata embedded in the PDF. The second is vision-model-based solutions (e.g., Microsoft LayoutLM, Amazon Textract, Adobe PDF Extract API), which render pages as images and use deep learning to identify regions such as headings, paragraphs, tables, and columns. For natively digital PDFs, rule-based methods are typically sufficient and significantly faster. For scanned documents or complex layouts, vision-model approaches offer meaningfully higher accuracy, at the cost of increased inference time and latency. In RAG systems, tables in particular deserve dedicated treatment — converting tables to Markdown format or structured JSON before chunking yields substantially higher retrieval accuracy than flattening table text into a continuous string, because it preserves the semantic row-column relationships that allow the model to correctly understand "which value belongs to which field" when generating an answer.

Where Real-World Scanned Document OCR Fails

Interestingly, digitally generated PDF text layers are generally clean and straightforward to extract. Real-world scanned documents are where OCR becomes a nightmare:

• Handwritten annotations mixed in with printed text;
• Low-quality, skewed, or smudged scans;
• Checkboxes, signature fields, and other form elements;
• Tables that break across pages.

These are precisely the scenarios most likely to fail silently in production — and the blind spots hardest to account for in an idealized architecture. When evaluating OCR solutions, teams should build dedicated test sets using this kind of "dirty data" rather than validating only against clean, natively digital PDFs.

From Ideal Architecture to Ugly Edge Cases

Building a reliable RAG system isn't about pursuing a perfect architecture — it's about developing a deep understanding of what actually breaks in production. The following core questions form a production-readiness checklist for RAG:

1. How do you detect and maintain document identity across versions?
2. What happens when a section is renamed, moved, split, or merged?
3. How do you prevent stale embeddings from silently dominating retrieval?
4. Where do scanned document OCR and layout extraction typically fail?
5. What failure cases or test documents do you use for validation?

Far too many technical write-ups stop at "how to build a RAG demo" and sidestep the engineering reality of making a system work reliably on real, messy, constantly evolving data. Collecting genuine edge cases and continuously stress-testing your system against them is the right path toward production-grade RAG.

Closing: The Real Test of a Provenance System

Building a RAG system that can answer questions isn't hard. Building one that can accurately explain "which version of which document did this answer come from" — that's the real challenge. Document version control and scanned document processing are exactly the two weak points that surface first when provenance capabilities are put to the test in practice.

For teams building enterprise knowledge bases, compliance systems, or contract management tools, these aren't optional concerns — they're the baseline that determines whether a system can be trusted. Rather than chasing elegant architecture diagrams, go collect those "ugly edge cases" and use them to continuously interrogate your system. Because in production, those edge cases are what decide whether you succeed or fail.