Footnote: Sentence-Level Citation Agentic RAG That Makes Every Answer Verifiable

"Chat with your documents" apps typically search once using the user's own words and then hope they got it right. A question with two parts only triggers one retrieval, so you get half an answer. Footnote replaces this fixed pipeline with an Agent that works like a researcher, and it has just one iron rule: an answer that can't be verified isn't an answer.

How It Differs from Ordinary RAG

Traditional RAG is single-round retrieval: throw the question into a vector store, grab a few chunks, stuff them into context, and let the model generate. The problem is that the model "hand-writes" its citations — it types out its own citation numbers, and the retrieval system never verifies whether that passage actually came from that location.

Footnote's approach is to return retrieval results as sentence-level blocks, with citations verified by the API rather than typed in by the model. So every citation maps to a locatable, highlightable source sentence. This small design change directly determines the "verifiability" of the answer.

RAG (Retrieval-Augmented Generation) is the mainstream architecture for "chat with your documents" apps today: first retrieve passages relevant to the question from a corpus, then feed those passages as context to an LLM to generate an answer. A common engineering practice is to split documents into fixed-length chunks (e.g., 512 tokens) and vectorize them into a store.

Sentence-level blocks are a refinement of chunk granularity. Traditional chunks often span multiple sentences or even paragraphs, so when a model cites them, it can only indicate "this passage comes from some chunk" without pinpointing the specific sentence. Sentence-level blocks make each record returned by retrieval a single sentence, with a stable ID and positional information, so a citation can be anchored to that unique source sentence. Finer granularity usually makes recall results more fragmented, so fusion and reranking are needed to keep context intact.

Footnote's Four Core Capabilities

1. It Plans Its Own Retrieval Rhythm

When the Agent receives a question with two parts, it splits it into two retrievals and organizes query terms using the document's own phrasing — rather than copying the user's colloquial language. Every retrieval step is displayed in real time in the sidebar, making the research process transparent and observable.

Behind this, Claude is given two tools: search_handbook and read_page, then called repeatedly in a loop until it can answer. If results are too thin, it searches again — basic researcher habits that a fixed pipeline can't achieve.

An Agentic Loop refers to the model advancing a task through back-and-forth cycles of "think — call tool — observe result — think again" until a termination condition is met. The two tools here — search_handbook for recalling candidate sentences by query terms, and read_page for reading the full content of a specific page — form a division of labor of "coarse search first, then close reading." The model can call them multiple times within a single answer, for example rewriting the query and searching again when the first retrieval is thin, or reading through the page containing a matched sentence to confirm context.

Compared with fixed "retrieve once, then generate," this loop hands the judgment of "are results sufficient?" to the model itself. The cost is increased token consumption and latency, so real systems typically need to set maximum rounds, timeouts, and budget caps to avoid endless retrieval.

2. Citations Precise to the Sentence

Click a citation chip, and the source document scrolls to the corresponding position and highlights the cited sentence.

According to the author, citations in this pipeline aren't "self-reported" by the model but verified by the API based on sentence-level blocks. For delivery scenarios — employee handbook Q&A, compliance document queries, internal knowledge bases — this matters far more than answer fluency, because a single misplaced citation destroys the credibility of the entire answer.

3. If It Doesn't Know, It Says So

If after retrying there's still no relevant content, Footnote tells you directly that the documents don't cover the question, rather than fabricating a plausible-looking answer.

You can see its handling in the screenshot: when asked about something the handbook doesn't cover, it explicitly states this while citing the closest passages so you can judge where the boundary lies. This "honest sense of boundaries" isn't common in RAG systems, but for any research tool meant to be actually used, it's the baseline.

4. Retrieval Methods Are Swappable

Footnote supports switching between hybrid, keyword-only, and vector-only retrieval modes at runtime, and displays each passage's ranking under both retrieval methods. For developers wanting to study the impact of retrieval strategies, this is a ready-made comparative experiment bench.

The three retrieval modes each have their strengths. Keyword-only (e.g., BM25) relies on term matching and term frequency statistics — strong at recalling precise terms like proper nouns, IDs, and abbreviations, but unable to handle synonymous paraphrasing. Vector-only retrieval encodes sentences into dense vectors and ranks by cosine similarity, capturing semantically similar expressions with different wording, but tends to miss rare terms. Hybrid retrieval runs both paths simultaneously, then merges rankings with a fusion algorithm.

RRF (Reciprocal Rank Fusion) is a common fusion method: take the reciprocal of each result's rank in both paths and sum them (the weighted version multiplies each path by a weight), then rerank by total score. It relies only on rankings, not on specific scores, making it naturally suited to keyword and vector scores with different scales. The tuning in this article on a 59-question evaluation set was mainly calibrating this set of weights and fusion parameters.

Tech Stack and Implementation Details

• Language and framework: Python + FastAPI, with a single-file HTML UI.
• Model calls: Direct calls to Claude via the Anthropic SDK, without any Agent framework.
• CLI: Prints every tool call, convenient for debugging and observation.
• Retrieval layer: Hand-written BM25 + local bge-small vectors, fused with weighted reciprocal rank fusion, and tuned on a 59-question evaluation set.
• Demo corpus: The public-domain 18F employee handbook, 232 pages total.
• Demo mode: Screenshots come from a key-free demo where model turns are scripted, but retrieval and citations run for real.

Source code is on GitHub at IbrarYunus/footnote.

bge-small is the small-size version of the BGE series of sentence embedding models released by BAAI, with roughly tens of millions of parameters and 384-dimensional output vectors, often deployed locally for latency- and VRAM-sensitive scenarios. After encoding a sentence into a dense vector, retrieval reduces to nearest-neighbor vector search. In this project, the vector store runs entirely locally with no external hosting service, so the demo requires no online inference.

Notably, the author deliberately avoided Agent frameworks like LangChain or LlamaIndex, instead orchestrating the loop directly with the Anthropic SDK. The benefit is a fully controllable pipeline: every tool call, every retrieval result, every citation ID can be printed in the CLI, making debugging and reproducing experiments easier. For a system whose core goal is "verifiability," reducing black-box indirection layers matters more than speeding up development.

Why This Direction Is Worth Watching

There's a key line from the author in the original article: retrieval is the easy half; the hard part is making every sentence of the answer verifiable.

This pinpoints a common weakness in current RAG applications. Most teams spend their energy on recall rates, chunk sizes, and embedding models, yet assume "model-generated citations" are trustworthy. Footnote flips this, placing the engineering difficulty on citation verifiability, using sentence-level blocks and API validation as a safety net.

This also explains why the Agent loop is necessary here: a single retrieval can't judge "are results sufficient?", but a loop that can retry and say "I don't know" can. Combined with the swappable retrieval strategy design, Footnote is more of a developer-facing research template than a finished product. Anyone who's been burned by "fabricated citations" in document Q&A will probably want to see how it's done.

One-line Summary

Footnote uses an Agent loop and sentence-level citation validation to make every RAG answer verifiable.

Paragraph Summary

Footnote is a sentence-level citation Agentic RAG project that replaces the traditional "single-round retrieval + model-written citations" pipeline with an Agent loop that plans retrievals, retries, and admits when it doesn't know. It uses two tools, search_handbook and read_page, to retrieve repeatedly with queries organized in the document's own wording; citations are validated by the API based on sentence-level blocks, so clicking one highlights the source sentence, rather than a model-typed number. When documents genuinely don't cover a question, it clearly states the boundary rather than fabricating an answer. The retrieval layer is hand-written BM25 plus local bge-small vectors, fused with weighted RRF and tuned on a 59-question evaluation set, with three swappable retrieval modes. The author's core judgment: retrieval is just the easy half; making every sentence of the answer verifiable is the hard half.