HandbookBrain: A Graph Database Brain That Makes Every Company Rule Traceable

When a new employee asks "why did trial MQLs drop at the end of July," she may have to dig through forty chat threads, meeting recordings, multiple revisions of a document, and CRM fields, and still can't explain "who decided what, and why." This is exactly the problem HandbookBrain aims to solve: it builds the decision paths behind company rules into a traceable, versionable knowledge graph.

Decisions Move Faster Than Documentation

Business rules are always changing: lead scoring thresholds get adjusted, Deal Desk adds new checks, partner renewal rules get reversed. Decisions happen in meetings or Slack threads, but documentation may not be updated for weeks, or never. Answers, documents, CRM fields, and dashboards that rely on old rules quietly go wrong. Traditional search can only find a document, but what you really need is the path: meeting → decision → decision maker → the commit that implemented the change → the changed section version → everything that depends on it. This is a graph problem, so it should be solved with a graph.

HandbookBrain maintains a versioned knowledge graph on FalkorDB, containing company documents, meetings and chats, merge requests that implement decisions, and the systems, CRM fields, metrics, and dashboards that documents touch. An agent works on top of this, answering "how does X work, and why," citing the exact version of the section used, with leads back to the meeting, decision maker, and merge request.

Data Model: Three Key Design Choices

The entire design revolves around three ideas.

Version by section, not by document. Pages are split into sections, and each change creates a new Section node that points to the old version via SUPERSEDES, with valid_from/valid_to. Nothing is overwritten, so the graph can be read "as of any date," and deleted rules remain queryable.

Answers are nodes. An Answer node CITES the exact section version it used. It is this edge that allows later changes to find the answers they break.

Decisions are first-class citizens. Each decision is extracted from meetings, chats, or tickets, with speaker and verbatim quote, linked to relevant sections, and reconciled with implementation commits by reading diffs. The core relationships can be simplified as:

Conversation <-DECIDED_IN- Decision -DECIDED_BY-> Person
Decision -IMPLEMENTED_BY-> Commit <-MERGED_AS- MergeRequest
Commit -CHANGED-> Section -SUPERSEDES-> older Section
Page -HAS_SECTION-> Section -MENTIONS-> System | Field | Metric | Dashboard
Answer -CITES-> Section     Answer -ANSWERS-> Question
Answer -INVALIDATED_BY / REVALIDATED_BY-> Commit

SUPERSEDES chains and point-in-time queries: SUPERSEDES is a directed edge from a new Section to the old version it replaces, and together with the valid_from/valid_to timestamp pair, it forms a traceable version chain. To read the graph "as of any date," you simply filter for nodes where valid_from <= D AND (valid_to IS NULL OR valid_to > D) to reconstruct the sections that were in effect at that time; a deleted rule corresponds to a Section whose valid_to is closed and has no successor, so it can still be retrieved and cited. This "append-only, never overwrite" storage approach leaves an immutable audit trail for every change, and also provides the graph structure foundation for the later traversal of "which answers did this change break."

Answering "Why" and "What Broke" with Cypher

"Why is it this way?" is a path from a section back to a meeting. The query starts from Section, goes through Commit and Decision to find Conversation, and optionally returns the decision maker's name and implementation time.

"Which answers did this change break?" starts from the commit, rolls back to the version cited by each answer, and returns the old and new text for a diff-based judge. The key handling is: if the change is "removed," then the current section itself is cited; otherwise, the previous version is cited.

"What else depends on this change?" goes from the change to the fields and metrics it mentions, then to all other current documents and their owners. The condition is o.valid_to IS NULL and a different path, thereby finding pages that are still in effect but share the same system entity.

Retrieval is hybrid, all within FalkorDB: a vector index of section embeddings, plus a full-text index of titles and body text, fused with reciprocal rank fusion, and filtered to the current version. Vectors, full text, and graph traversal are in the same database, meaning there is no second store to synchronize. The agent is a tool-calling loop wrapping 10 parameterized read-only Cypher tools; every assertion must carry a citation actually returned by a tool, checked by code. A reviewer running on a different model sees only the question, evidence, and draft, and can send it back for rewriting.

Reciprocal Rank Fusion (RRF): Hybrid retrieval merges the ranking results of vector similarity and full-text search. RRF is a commonly used parameter-free fusion method. It takes 1/(k + rank) for each document's rank in each result list (k is usually 60), sums them, and re-ranks by total score. Compared with directly adding scores of different scales, RRF depends only on rank and requires no normalization, robustly balancing semantic recall and exact keyword matching. In this project, the vector index of section embeddings and the full-text index of titles/body text each produce a ranking, which are fused with RRF and then filtered to the current version where valid_to IS NULL, ensuring retrieval results are both semantically relevant and version-valid.

Maya's Question, Answered in One Go

In the replay demo, the trial threshold story unfolds like this: on May 21, real MR !19600 added "Trial Threshold Scoring" to the Marketo page, and HandbookBrain linked it to a (synthetic) marketing operations meeting 13 days earlier. On June 1, a new analyst asked "how many points make a trial user an MQL?" The answer was 25 points, citing that section version. On July 17, a (synthetic) marketing operations weekly meeting decided to raise the threshold. On July 23, real MR !20534 changed 25 to 31. The June 1 answer was judged stale by the diff, the owner in CODEOWNERS was notified, and the answer was refreshed to 31, with the meeting, decision maker, and MR attached. Maya asks one question and gets the entire path.

Auditing the Graph with Independent Sources

An answer engine that cannot be verified is a liability. The author wrote an audit that checks what the site displays against sources the agent never reads. Change dates and MR numbers, version chains were checked against git log, 2562/2562 passed; decision makers, locations, and quotes were checked against meeting records, 21/21 passed; document owners were checked against CODEOWNERS, 122/122 passed; decision-to-implementation-MR links were checked against hidden labels, 0 bad links. The audit also caught real problems: the initial decision-to-MR judge was too lenient, linking three decisions to later unrelated changes; now it only recognizes what actually changed in the diff.

Evaluating the Agent Honestly

The author's initial headline number was "89% vs 0%" for "why" questions against a plain RAG baseline. It looked great but was unfair: after fact-by-fact checking, plain retrieval was just as good as the agent at finding meeting dates and decision makers; it scored 0% only because merge requests were not in its data at all.

So the evaluation was rebuilt following Anthropic's "Demystifying evals for AI agents": use code graders wherever possible, and model judges only for open-ended answers; include questions that should be answered and questions that should be refused; 3 trials per question, with LLM caching off. A stronger baseline, "RAG + git log," was also added, with the full merge log text.

Results for 29 questions × 3 trials: HandbookBrain overall 87%, RAG + git log 66%, plain RAG 59%. On "name the MR that implemented the decision," HandbookBrain scored 18/18, RAG + git log only 4/18, plain RAG 0/18. On "old/new values of a changed rule," HandbookBrain 15/15, RAG + git log 9/15, plain RAG 6/15. Meeting dates were 18/18 for all three; plain lookups were 70% for all three.

These numbers show: finding the meeting is a retrieval problem, and retrieval can solve it. What the graph really adds is "the link from decision to implementation change" and "version history," including deleted rules. Even with the entire merge log in context, the model rarely matches a meeting to the correct MR on its own. On plain lookups, the graph doesn't help, and the agent is slower and more expensive. The author also admits limitations: questions come from a single dataset, and the agent was fixed after seeing failures on these questions, so numbers on new questions will be lower.

The evaluation also found real bugs: a section called "Trial Threshold Scoring" was buried under a generic "MQL and Lead Scoring," causing the agent to answer 100 instead of 31; the fix was to add title-match candidates to search. Wide Markdown tables were misread by columns; the fix was row linearization. Rules removed from a page were invisible; the fix was for search to also return older versions that said more, plus the MR that deleted them. Rules agreed only in Slack were treated as policy with a handbook citation; the fix was to mark them as "only said in chat," which the agent must state.

LLM-as-judge and code graders: Open-ended answers are hard to score with exact matching, so a common approach is to have another model judge whether an answer is correct and cites sufficient evidence according to a rubric. But model judges have position bias, verbosity preference, and self-consistency bias, so Anthropic's advice is: anything that can be verified with deterministic code (such as numbers, MR IDs, whether citations are included) should always use code graders; only truly open-ended phrasing goes to a model judge, with a fixed judge model, reference evidence provided, and multiple samples taking the majority when necessary. The article's "3 trials per question, LLM caching off" is precisely to reduce randomness and make scores reproducible, avoiding cache hits masking the agent's true variance.

What Was Learned

Make answers nodes. Once an answer cites an exact version, "what did this change break" is a traversal, not a guess. Version at the section level. Whole-document versioning is too coarse to determine which answer a change affects. Measure fact by fact, and against a fair baseline. Aggregate scores hide where the graph helps and where it doesn't; an honest table is more convincing than a pretty one. Audit LLM-built edges with independent sources. The model that extracts links should not also be the judge.

Next Steps

Direct connectors (Slack, Zoom, Drive) to replace exports, and speech-to-text for recordings; make CRM and BI metadata (Salesforce fields, Power BI / Tableau dashboards) first-class nodes; open merge requests directly for approved updates instead of just recording sign-off. The project is built with FalkorDB, Python, FastAPI, and LiteLLM, and most of the code was written by the AI coding assistant Claude under the author's guidance and review.

If your team's rules are scattered across five tools and no one remembers why they exist, this approach is worth considering: put decisions, implementations, and documentation in the same graph, and make "why" a path you can walk.

One-line Summary

HandbookBrain uses a versioned knowledge graph to trace the decision paths behind company rules, and evaluates its effectiveness honestly.

Paragraph Summary

HandbookBrain addresses the difficulty of tracing "why company rules are the way they are" by building a versioned knowledge graph on FalkorDB: versioning by section, modeling answers as nodes that cite exact versions, and treating decisions as first-class citizens linked to meetings, decision makers, and implementation commits. The agent performs hybrid retrieval across vectors, full text, and graph traversal through 10 read-only Cypher tools, and every assertion must carry a citation returned by a tool. An independent audit verified change dates, decision citations, and owners against git log, meeting records, and CODEOWNERS. In a 29-question × 3-trial evaluation, it scored 87% overall, outperforming RAG + git log at 66% and plain RAG at 59%, with its advantage concentrated in decision-to-MR links and version history; the author also acknowledges the evaluation's limitations and used failure cases to drive several fixes.