Building a Hindi Scanned-Document RAG Assistant: OCR and Hybrid Retrieval in Practice

Most RAG demos start with a pile of English PDFs, but real-world documents are often trickier. Recently, developer Abhishek shared his experience building a knowledge assistant for SVSU University—starting from a batch of scanned circulars in Hindi and English that had been photocopied, scanned at an angle, and were full of ruled tables, and university staff needed to find answers in them quickly. His practice reveals an often-overlooked fact: in a RAG system, retrieval and models matter, but OCR quality determines everything, and vector retrieval alone is far from enough for exact queries like codes and dates.

OCR Quality Determines Everything

Abhishek spent his first few weeks focused on retrieval and models, but soon discovered that if the OCR output text is fragmented or wrong, nothing downstream can be done well. He summarized three key points that changed everything:

• Detect text type per page: Before running Tesseract, determine the page language and select the appropriate language model. Some pages mix Hindi and English; his approach was to run both and take the result with higher confidence.
• Rebuild tables cell by cell: Most circulars are tables—fee schedules, academic calendars, code lists. Raw OCR output turns tables into a tangled mess. By detecting table lines and reading each cell individually, "course code 302" can be correctly associated with "fee ₹1,200" instead of being scattered across three lines.
• Flag low-confidence words: Let reviewers see uncertain words instead of silently guessing.

This pipeline runs in the background: staff upload scans and then go on with their work.

Tesseract is an open-source OCR engine that supports over 100 languages, including Hindi (language code hin) and English (eng). In mixed-language documents, it is usually necessary to run different language models separately and compare results. Each word output by Tesseract comes with a confidence score, which can be used to flag low-confidence words for human review. In addition, Tesseract has limited table recognition capability and often parses table structure as linear text, so additional table line detection and cell reconstruction steps are required.

pgvector is an extension for PostgreSQL that provides storage, indexing (IVFFlat, HNSW), and similarity search operators for vector types (such as vector(768)). It allows structured data (page numbers, bounding boxes) and vector embeddings to be stored in the same database, simplifying the architecture of a RAG system. During queries, the <=> operator computes cosine distance, and combined with filter conditions it enables hybrid retrieval.

Cross-encoder reranking is a common technique for two-stage retrieval: first, vector retrieval or keyword matching recalls candidate passages, then a cross-encoder (such as cross-encoder/ms-marco-MiniLM-L-6-v2) finely scores each "query-passage" pair. Unlike bi-encoders, a cross-encoder sees both the query and the passage at the same time and can capture finer-grained relevance, but its computational cost is high, so it is only suitable for reranking a small number of candidates.

Storing the Exact Location of Every Passage

Every passage and every table row is embedded and stored in PostgreSQL using the pgvector extension. At the same time, the system also saves the page number and the source bounding box. This seems like extra work, but the benefit is clear: answers can cite "Circular 42, page 3, line 7," and clicking the citation opens the scan and scrolls to and highlights the exact location. For people who need to act on the answer, the citation itself is the product.

The Blind Spots of Vector Retrieval and Hybrid Retrieval

Semantic search performs well for questions like "What is the policy on late fees?" but completely fails for "What does code EX-2024-17 say?" Circulars are full of codes, dates, and exact amounts, and these are precisely what users actually search for. His solution was to improve the retrieval pipeline:

1. Rewrite the question into a query and a Hindi query (users speak Hinglish, and documents may be in either language).
2. Run vector search to find similar passages.
3. Match codes, dates, and rare words.
4. Merge the results and rerank them with a cross-encoder model.

Abhishek said that the reranker was the single biggest improvement in the entire project.

Answer Based on Passages, Then Verify the Answer

After the language model receives the retrieved passages, it is asked to answer based only on those passages and to attach numbered citations. Then a faster model verifies the answer against the source documents and flags any unsupported claims. Such errors are rare, but they are exactly the kind that destroy trust.

Answers are presented in a streaming fashion, displayed as they are written, which makes the whole experience feel fast even if retrieval takes a second.

The Boring Parts Are the Product

What really makes this assistant usable is a series of seemingly boring engineering details:

• Draft-to-active approval workflow: Documents cannot be used before review.
• Roles and document-level visibility: Some circulars are only for faculty and staff.
• Automatic expiration: Circulars with end dates are automatically retired.
• Two-step verification and locking: This is university data, so security matters.
• Automated answer quality tests: A set of questions with known answers runs in CI on every push, and if retrieval degrades, the deployment fails.

CI/CD runs through GitHub Actions to AWS servers, using Docker and Caddy. Nothing fancy—and that is exactly the point.

One API, Two Clients

The web app is built with Next.js, and the Android app with Flutter. They share the API and the same set of accounts. The mobile app can photograph paper letters and display in-app update notifications. Widget tests open every screen in a phone viewport in CI. Because the API was already solid, building the second app took only a fraction of the time of the web app.

What He Would Do Differently

Abhishek gave three lessons:

• Start with an OCR evaluation set, not a chat UI.
• Tie chunking to document structure (lines, sections) from day one, rather than using fixed windows.
• Reserve time for the approval workflow early, because it affects the data model more than the AI does.

This case shows that when building RAG assistants for real-world documents, the effort is often outside the model—OCR, structured storage, hybrid retrieval, and engineering details are what make the system trustworthy.

One-line Summary

Hindi scanned-document RAG in practice: OCR quality, hybrid retrieval, and engineering details determine success.

Paragraph Summary

This article shares experience building a Hindi/English scanned-circular RAG assistant for SVSU University. The core lesson is that OCR quality determines everything: detect language per page, rebuild tables cell by cell, and flag low-confidence words. Storing the page number and bounding box of every passage enables precise citations and highlighting of the original text. Vector retrieval cannot handle exact queries like codes and dates, so hybrid retrieval is needed (vector + keyword matching + cross-encoder reranking), with reranking providing the biggest improvement. Answers are generated by a language model based on passages and then verified by a second model. Engineering details such as approval workflows, role permissions, and automatic expiration are crucial to product usability.