No Vector Database Needed: Hand-Writing Cosine Similarity with fastembed + numpy

Many RAG tutorials start by pushing you toward a vector database, a chunking framework, and an embedding API. But for small corpora like a personal résumé, these aren't architectural necessities—they're extra maintenance burden. While building JobCopilot, the author of this article generated embeddings with fastembed and did retrieval with numpy. The whole pipeline is about 60 lines of code, and along the way they documented how a TOP_K=12 default copied from a tutorial quietly broke retrieval quality.

1. Building the Index: Chunking and Embedding in One Pass

The indexing side is astonishingly little work. The author wrote a one-off script that: splits the résumé by ## headings using a regex, treating each section as a chunk; generates 384-dimensional vectors with the BAAI/bge-small-en-v1.5 model (about 33MB on disk, runs purely on CPU, no API key needed); and normalizes the vectors at build time before storing them in a JSON file.

• re.split(r"\n(?=## )", md) chunks everything with a single regex—headings stay in the body, and empty headings are dropped.
• The model is lazily loaded, only pulled in on first use, to avoid import-time overhead.
• Normalization happens at build time rather than query time so that the runtime cosine similarity collapses into a single dot product.

The author's key mental shift: "If one command can regenerate the index, then it's not infrastructure—it's a build artifact." That takes the mystique out of RAG.

2. Runtime Retrieval: 10 Lines of Code, One Matrix Multiply

The query side is the heart of the retrieval engine, and the code is minimal: load the JSON store, embed the query text and normalize it, then run scores = emb @ q, and take the top K with argsort. emb has shape (n_chunks, 384) and q is (384,), so a single matrix multiply broadcasts the query across all chunks.

The case for skipping a vector database is straightforward: with a single-digit number of chunks, the entire "index" is just a matrix, and a full scan is one matrix multiply taking microseconds. Approximate nearest-neighbor indexes only show their speed advantage at hundreds of thousands of vectors, at the cost of recall error and an extra service. At personal-résumé scale, the tradeoff is inverted—complexity buys you speed you don't need.

The author tested the query "Kubernetes, Terraform, SRE on-call" against a six-chunk demo résumé, and the scores showed clear separation: the relevant chunk at 0.784, the rest between 0.573 and 0.487. That shape—the relevant chunk rising out of the noise band—is exactly the precondition for retrieval being useful.

3. The Tutorial Trap: An Oversized TOP_K Degrades Ranking into Bulk Loading

The author admits the first version used TOP_K = 12 because that's what the tutorial did. It seemed "more robust"—grab more and let the LLM judge relevance. But the effect was the opposite.

The failure is silent: no crash, no warning. argsort()[::-1][:12] just returns twelve chunks: one that genuinely matches the job description, plus eleven that match the general register of "résumés and job descriptions"—Python keywords, generic résumé boilerplate. This tail noise gets stuffed into the prompt, and the LLM confidently writes it into a missing-skills list. Garbage in, confident garbage out.

The score line reveals the problem: on the real store, the top score is 0.635 and the tail is only 0.450. Within that 0.185 gap, only the first line deserves attention; everything below 0.55 is the noise band. Retrieval is no longer ranking—it's bulk loading with more steps.

The fix is one line: TOP_K = 8. After the change, the top score is 0.635, the second line 0.592, and the language/education chunks at the end of the list score below 0.45—visible separation reappears at the top.

The author distills a rule: TOP_K must be small enough to keep a visible gap between the top score and the rest. Once the gap disappears, retrieval becomes decoration and the prompt becomes a firehose. Tutorials won't tell you this—perfectly correct code can still be wrong because the constants you copied were tuned for someone else's scale.

4. From Ranked Chunks to Prompt: Character Budget and Whole-Chunk Truncation

Retrieval returns ranked chunks with scores, but the prompt needs a single string. Naive concatenation has two failure modes: one excessively long section eats the prompt's attention budget; and a [:2400] slice cuts a chunk in half, leaving half a bullet list floating in the prompt.

The build_context function solves both with two details:

• Stop once max_chars is reached, even if that means using fewer than TOP_K chunks. This is a prompt-quality knob, not a storage quota: 2,400 curated characters beat 6,000 characters of everything.
• Truncate at chunk boundaries. Chunks that don't fit are skipped entirely—no half-content.

The author also tested the query prefix mentioned in the bge model docs: "Represent this sentence for searching relevant passages:". The result was the top score rising by 0.014, but the gap to second place actually shrank—wasted effort for ranking. So this pipeline has no query-prefix engineering; normalization is the only trick worth keeping.

5. Lessons from Small-Scale Retrieval

The whole pipeline—chunk by heading, embed once with a 33MB model, normalize at build time, emb @ q at runtime, take the top 8, concatenate under a character budget—is about 60 lines of code, and the author understands every line. That's worth more than a vector database they don't need yet.

Three transferable lessons:

1. For small corpora, the "vector store" is a JSON file plus one matrix multiply. Tutorials won't tell you when to reach for something heavier—your own score gap will.
2. Copied defaults carry the original author's scale. TOP_K = 12 was fine in the tutorial's corpus but quietly harmful in the author's résumé.
3. The most honest metric isn't the top score—it's the gap between the top score and the runner-up. Once the gap is gone, nothing downstream matters.

For developers building small RAG applications, this approach offers a runnable baseline: get hand-written cosine similarity working first, and only consider introducing a vector database when your score gaps and scale genuinely call for it.