Q&A Chunking for RAG in Practice: Design Insights for a Tax Compliance Assistant

Project Background

A developer shared a practical case study of building a RAG (Retrieval-Augmented Generation) chatbot for Indian GST (Goods and Services Tax) compliance. The project is based on official government FAQ documents and aims to provide traceable, accurate answers to tax-related questions. Unlike conventional RAG systems, this project implemented targeted optimizations in its document chunking strategy.

RAG is an architectural paradigm proposed by Meta AI's research team in 2020 that combines information retrieval with text generation. Traditional large language models (LLMs) rely solely on parametric knowledge learned during training to generate answers, which easily leads to "hallucinations"—generating content that appears reasonable but is factually incorrect. RAG mitigates this problem by introducing an external knowledge retrieval step before generation: it first converts the user's question into a vector representation, retrieves the most relevant document fragments from a pre-built knowledge base, and then passes these fragments as context to the LLM for answering. This architecture enables models to generate answers based on actual document content, making it particularly suitable for professional domain applications that require factual accuracy.

Notably, the inherent complexity of India's GST system is the core driving force behind such tools. GST was officially implemented on July 1, 2017, replacing multiple taxes at both central and state levels (such as VAT, service tax, excise duty, etc.). The system involves three types of taxes: CGST (Central GST), SGST (State GST), and IGST (Integrated GST), with rates ranging from 0% to 28%, along with various exemptions and special provisions. The Indian government has published extensive FAQ documents, circulars, and rulings to explain various rules, which are frequently updated and contain detailed provisions. For small and medium-sized business owners and tax practitioners, quickly and accurately finding applicable provisions is a significant pain point.

Core Innovation: Semantic Unit Chunking

Why Abandon Fixed-Size Chunking

Traditional RAG systems typically use fixed-size text chunks (such as 512 or 1024 tokens), which has obvious drawbacks: complete answers may be forcibly split across different chunks, resulting in broken semantic integrity. The developer chose a chunking method more aligned with content structure—extracting each FAQ as an independent Q&A pair and treating it as a complete semantic unit for vectorization.

From a technical perspective, text chunking is a critical component affecting retrieval quality in RAG systems. Fixed-size chunking mechanically splits text by token count, which is simple to implement but has poor semantic awareness. The industry has developed multiple alternative approaches: recursive character splitting (recursively splitting by paragraphs, sentences, etc.), semantic similarity-based chunking (detecting semantic transition points for splitting), and structure-based chunking as adopted in this project. The core idea of structured chunking is to use the document's native organizational structure—FAQ question-answer pairs, legal text clauses, API documentation endpoint descriptions—as chunk boundaries. This method preserves the author's intent in organizing information, making each chunk a self-contained knowledge unit that more easily matches user intent precisely during retrieval.

This approach brings three key advantages:

Semantic Integrity: Questions and answers naturally pair together, avoiding context fragmentation

Retrieval Precision: User questions match FAQ questions more directly

Interpretability: Each retrieval result is a complete knowledge unit

Technology Stack Selection

The project uses the BGE-M3 model for text embedding, which is a multilingual, multi-granularity embedding model particularly suitable for cross-lingual and domain-specific document retrieval scenarios. BGE-M3 was developed by the Beijing Academy of Artificial Intelligence (BAAI), and the three M's in its name represent Multi-Linguality, Multi-Functionality, and Multi-Granularity. The model supports over 100 languages, can simultaneously generate dense vectors, sparse vectors, and ColBERT-style multi-vector representations, with a maximum input length of up to 8192 tokens. It performs excellently on multilingual benchmarks MTEB and MIRACL. For the Indian tax scenario, BGE-M3's multilingual capability is particularly important—Indian official documents may mix English with local languages like Hindi, and traditional monolingual embedding models often perform poorly when handling such mixed-language documents.

For vector storage, Qdrant is used, which is a high-performance vector database supporting efficient similarity search and flexible filtering mechanisms. Written in Rust, Qdrant is known for its memory efficiency and query speed compared to similar products like Pinecone, Weaviate, and Milvus. It supports multiple distance metrics (cosine similarity, Euclidean distance, dot product, etc.), provides rich filtering capabilities (can apply metadata condition filtering during vector search), and supports sharding and replication for horizontal scaling. Qdrant can be deployed as a standalone service or run in embedded mode through a Python client, and this flexibility makes it particularly suitable for smooth transitions from prototype to production.

Quality Control Mechanisms

Similarity Threshold Filtering

The developer added similarity threshold checking before LLM processing. Only when the similarity between retrieved FAQs and user questions exceeds the set threshold are they passed to the large language model. This mechanism effectively prevents interference from irrelevant content and improves answer reliability.

This design has important engineering considerations. In RAG systems, vector retrieval always returns the "most similar" results, but "most similar" does not equal "similar enough." When the knowledge base does not contain content related to the user's question, the retrieved results may be completely irrelevant but still passed to the LLM. Upon seeing these irrelevant contexts, the LLM may force-fit and generate seemingly reasonable answers, causing serious misinformation. The similarity threshold mechanism essentially acts as a "gatekeeper"—by setting a minimum similarity score (e.g., 0.7), retrieval results below the threshold are filtered out. Threshold selection requires balancing recall and precision: too high may filter out valid answers, too low may introduce noise. In practice, tuning through evaluation sets is typically needed.

Mandatory Source Attribution and Rejection Capability

The system prompt explicitly requires:

• All answers must cite sources
• When retrieval results are irrelevant, must explicitly respond "I don't know"

The developer conducted adversarial testing by deliberately asking off-topic questions, and the system demonstrated good rejection capability. This is crucial in practical applications—a tax consultation system that makes things up could lead to serious legal and financial risks.

Practical Significance and Reflection

Design Philosophy for Domain-Specific RAG

This project reveals an important principle: RAG system chunking strategies should align with the natural structure of content. For FAQ documents, Q&A chunking is the natural choice; for technical documentation, chunking by sections or API entries may be more reasonable; for legal texts, chunking by clauses may be necessary.

Blindly applying generic fixed-size chunking may lose the structured information of documents and reduce retrieval effectiveness.

Open Source and Reproducibility

The developer open-sourced the complete code on GitHub and deployed an online demo on Hugging Face Spaces. This open attitude provides a referenceable implementation example for other domain-specific RAG applications.

Potential Improvement Directions

Although the project has already shown good results, there are still some optimization opportunities worth exploring:

Hybrid Retrieval Strategy: Combine keyword retrieval (BM25) with vector retrieval to improve recall and precision in different scenarios. BM25 is a classic frequency-based text retrieval algorithm proposed by Stephen Robertson and others in the 1990s and remains the core ranking algorithm for search engines like Elasticsearch. It evaluates relevance by calculating term frequency (TF) in documents, inverse document frequency (IDF), and document length normalization. Unlike vector retrieval which relies on semantic understanding, BM25 excels at exact matching of keywords and proper nouns. In tax scenarios, precise terms like "GSTR-3B," "HSN code," and "Section 16" are often more reliably retrieved by BM25 than pure vector retrieval. Hybrid retrieval runs both types simultaneously and merges results using methods like Reciprocal Rank Fusion (RRF), combining the advantages of semantic understanding and exact matching.

Dynamic Threshold Adjustment: Dynamically adjust similarity thresholds based on question complexity or user feedback to achieve better balance between accuracy and coverage.

Multi-Hop Reasoning Support: When a single FAQ cannot fully answer a question, attempt to combine multiple relevant FAQs for comprehensive answers. Multi-hop reasoning refers to complex questions that require extracting information from multiple sources sequentially or in combination. For example, if a user asks "Do service-based businesses with registered capital below 200,000 rupees need to pay GST," it may require retrieving and synthesizing multiple FAQs about "GST registration threshold," "service industry applicable rules," and "small-scale business exemption conditions." Common methods for implementing multi-hop reasoning include: iterative retrieval (generating new queries based on first-round retrieval results), graph structure retrieval (organizing knowledge as graphs and traversing relationship edges), and agent-based methods (letting LLMs autonomously decide whether additional retrieval is needed). Multi-hop reasoning significantly improves answering capability for complex questions but also increases latency and error probability.

User Feedback Loop: Collect user evaluations of answer quality for continuous optimization of retrieval and generation strategies.

Summary

This tax compliance assistant project demonstrates core design principles for domain-specific RAG systems: starting from the natural structure of documents, choosing appropriate chunking granularity; ensuring output quality through threshold filtering and prompt engineering; emphasizing source traceability and rejection capability. These principles apply not only to tax compliance scenarios but also provide valuable reference for other professional domain Q&A systems requiring high accuracy.