Hands-On RAG Project: A Guide to Building from Scratch and Benchmarking Retrieval Strategies

A Reddit user posted asking for help finding a real-world RAG project that could handle multiple formats like PDF, HTML, CSV, and Markdown, run the complete pipeline from data ingestion to generation, and compare the effects of different retrieval strategies. This need is quite representative—people learn the theory but don't know what data to use or what problem to tackle. This article addresses that demand by offering actionable project directions, public datasets, and engineering advice.

Why "Finding a Project" Is Harder Than "Learning a Framework"

Most RAG tutorials stop at the level of "load a PDF with LangChain, plug in a vector store, ask a few questions." Only when you actually start do you realize: when document formats get messy, parsing breaks down; when you switch retrieval strategies, all metrics change; without ground truth for evaluation, you can't even define "good performance." This user explicitly mentioned wanting to "experiment with the complete pipeline, from ingestion to generation, and benchmark different retrieval strategies"—which is exactly the part most tutorials skip.

A suitable practice project needs to meet several conditions: data is accessible and has real noise, the problem scope is clear, and retrieval quality is quantifiable. Below we cover three aspects: project directions, datasets, and benchmarking methods.

RAG Project Directions Worth Tackling

Technical Documentation Q&A Assistant

Pick an open-source project you're familiar with but that has extensive documentation (e.g., Kubernetes, Docker, or a database), and ingest all official docs, GitHub Issues, and Release Notes. The advantage of this data is that it's mixed-format (HTML docs, Markdown READMEs, CSV issue exports), naturally suited for testing a multi-format parsing pipeline. Questions are also easy to design—"how to configure persistent storage," "which API was deprecated in which version."

The key challenge is version conflicts: the same question has different answers in different versions of the documentation. This forces you to think about metadata filtering and time-aware strategies during retrieval, rather than relying solely on vector similarity.

Cross-Repository Academic Paper Retrieval

Use arXiv's public API to scrape paper abstracts and full texts in a specific subfield (e.g., RAG itself, or a medical area). PDF parsing is the main difficulty—formulas, tables, and two-column layouts all cause errors in general-purpose parsers. You can compare the actual performance of tools like PyMuPDF, pdfplumber, and Nougat.

Evaluation for this type of project can be very solid: given a paper's title, can the retrieval system hit it within the top 5 results? This is an objective metric that doesn't depend on subjective judgment.

Simulating an Enterprise Internal Knowledge Base

If you have access to your company's internal Confluence, Jira, or SharePoint data (after anonymization), use it directly for prototyping. This is the closest to a "real-world" scenario because real enterprise data is dirtier than you can imagine: outdated pages, duplicate content, permission isolation, abbreviations, and internal jargon.

If you don't have internal data, you can substitute with Notion's public template library or a specific topic from Wikipedia.

Metadata Filtering and Time-Aware Retrieval: In enterprise or technical documentation scenarios, each document typically carries structured information such as version number, publication date, author, and permission tags. Metadata filtering means using these fields to filter candidate documents before or after vector retrieval—for example, keeping only the latest version or retrieving only content the current user has permission to view. Time-aware strategies go further by dynamically adjusting weights based on the query's temporal intent—for instance, when a user asks "how to configure this currently," outdated versions should be downweighted. Both LangChain and LlamaIndex provide interfaces for metadata filtering, but the real challenge is designing reasonable filtering rules and version management schemes.

Public Datasets and Access Channels

Document-Type

• Wikipedia Dump: Large volume, uniform format (XML/Wikitext), suitable for testing large-scale ingestion and chunking strategies. The downside is that the content is too clean, lacking the noise of real documents.
• arXiv: Obtain via the official API or arXiv datasets on Kaggle. Both PDF and LaTeX source are available, enabling format comparison experiments.
• Common Crawl: Raw HTML, extremely realistic and extremely dirty. Suitable for testing cleaning and parsing pipelines, but not for quickly producing results.
• SEC EDGAR: Public company financial reports, mixed PDF and HTML, with clear Q&A scenarios (financial metric queries).

Q&A Evaluation-Type

• MS MARCO: Microsoft's Q&A dataset, with a large number of real queries and passage relevance annotations, can be directly used to evaluate the retrieval module.
• Natural Questions: Google's Q&A data, questions from real searches, answers are Wikipedia passages.
• BEIR Benchmark: A cross-domain retrieval evaluation benchmark, containing multiple datasets and standard metrics (nDCG, Recall). If you want to systematically compare retrieval strategies, this is the top choice.

Multi-Format Mixed Scenarios

The Reddit user specifically mentioned four formats: PDF, HTML, CSV, and Markdown. You can construct a mixed dataset yourself: take a CSV dataset from Kaggle, Markdown docs from GitHub, scrape HTML from some official website, and find a few public PDF reports. The advantage is controllability—you know the content of each document, making it easy to verify whether retrieval results are correct.

Practical Advice for Benchmarking Retrieval Strategies

Define Comparison Dimensions

Don't just compare "vector retrieval vs keyword retrieval." More meaningful dimensions include:

• Chunking strategy: Fixed length vs semantic chunking vs structure-based chunking
• Embedding models: Differences in retrieval hit rate for different models on the same data
• Hybrid retrieval: BM25 + vector + RRF fusion, compared to pure vector
• Reranking: Impact of adding or not adding Cross-Encoder reranking on final results

Semantic Chunking vs Structure-Based Chunking: Fixed-length chunking is simple but can cut off sentences or paragraphs. Semantic chunking computes embedding similarity between adjacent sentences and splits at semantic boundaries, preserving paragraph integrity. Structure-based chunking leverages HTML/Markdown markers like headings, lists, and tables to split content into logical units—for example, an H2 and its subordinate paragraphs as one chunk. The latter two methods generally improve retrieval quality but are more complex to implement. Additionally, RRF (Reciprocal Rank Fusion) is an algorithm that fuses ranked lists from multiple retrievers. It does not rely on score normalization and directly sums the reciprocal of ranks, commonly used in hybrid retrieval to merge BM25 and vector retrieval results.

Establish an Evaluation Pipeline

Use annotated data from BEIR or MS MARCO to compute Recall@k and nDCG@k. If there is no annotated data, you can use an LLM for automatic evaluation (letting the model judge whether retrieved passages contain the answer), but be aware of the bias. A more reliable approach is to manually annotate the correct answers for 50-100 queries as a small-scale gold standard.

Recall@k and nDCG@k: Recall@k measures whether the top k retrieval results contain all relevant documents, directly reflecting retrieval coverage. nDCG@k (normalized Discounted Cumulative Gain) considers the ranking positions of relevant documents, giving higher scores to higher-ranked ones, and normalizes across queries for comparability. These two metrics are the gold standard for retrieval evaluation, and the BEIR benchmark uses them. Cross-Encoder Reranking: Unlike dual encoders (which encode query and document separately), a Cross-Encoder concatenates the query and document and feeds them together into the model, capturing finer-grained interactions. This yields higher accuracy but slower speed, typically used to rerank initial retrieval results. Bias in LLM Automatic Evaluation: When using large models to judge answer relevance, the model may favor longer, more fluent passages, or have blind spots in specific domains, leading to scores inconsistent with human judgment. Therefore, small-scale manually annotated gold standards remain irreplaceable.

Record and Reproduce

For each experiment, record: data version, chunking parameters, embedding model, retrieval parameters, evaluation metrics. RAG experiments have too many variables; not recording means not doing. You can use MLflow or a simple CSV log.

Open-Source Projects You Can Contribute To

If you want to practice on existing projects rather than starting from scratch, consider:

• LlamaIndex and Haystack: These two frameworks' GitHub repositories have many "good first issue" labels, involving new document loaders and retriever implementations.
• RAGAS: A RAG evaluation framework that needs test data and use cases from various domains.
• BEIR: If you want to contribute new datasets or evaluation results, the community is very welcoming.

The benefit of participating in these projects is that you'll see edge cases raised by real users—things that never appear in tutorials.

Common Pitfalls and How to Handle Them

PDF parsing is the biggest time sink. Tables and formulas are almost guaranteed to cause errors. It's advisable to first run the full pipeline with simple documents, then gradually add complex formats. There is no universal chunk size; 256-512 tokens is a common starting point, but technical documentation may need chunking by heading hierarchy. The effectiveness of retrieval strategies is highly dependent on data characteristics—BM25 may beat vector retrieval on FAQs, while the opposite holds for semantically vague queries. Don't preset conclusions; let the data speak.

Finally, evaluation is harder than construction. Many RAG projects fail not because retrieval performance is poor, but because it's impossible to tell whether improvements are actually effective. Build the evaluation pipeline first, then optimize retrieval.

One-line Summary

An actionable guide on project directions, datasets, and benchmarking for RAG practice projects.

Paragraph Summary

This article responds to a Reddit user's request for a real-world RAG project, noting that "finding a project" is harder than "learning a framework" because most tutorials only cover simple pipelines. It recommends three project directions: technical documentation Q&A, cross-repository academic paper retrieval, and enterprise internal knowledge base simulation, emphasizing accessible data, clear problem scope, and quantifiable retrieval quality. For public datasets, it suggests Wikipedia Dump, arXiv, Common Crawl, and SEC EDGAR for documents, and MS MARCO, Natural Questions, and BEIR for Q&A evaluation. Retrieval strategy comparisons should cover four dimensions: chunking strategy, embedding models, hybrid retrieval, and reranking. The evaluation pipeline should use Recall@k and nDCG@k, and record all experimental variables. Finally, it warns that PDF parsing and chunking are common pitfalls, and evaluation is harder than construction—build the evaluation pipeline first.