A 1B Model Doesn't Read Your Letters: The Invisible Layer Between Documents and Local AI

When people discuss local AI, attention tends to focus on the model itself—parameter scale, inference speed, quantization precision. But one developer's practice reveals a neglected truth: what determines whether a small model can work reliably isn't how smart the model is, but how the content you feed it has been processed.

Before a Document Reaches the Model, It Isn't Yet Text

According to the developer, he built a tool that runs entirely locally, explaining Swiss administrative letters in plain Italian. The model is a 1B-parameter Gemma, running on CPU via Ollama; the tool is about 306 lines of Python 3.10 code, using only the standard library. The biggest lesson from the whole project has nothing to do with the model, but with the "unseemly" pipeline between the letter and the model.

A payment notice from a debt collection office might be a paper letter, or a PDF that behaves like paper: fixed layout, stamps, oddly placed reference numbers, and a table where amount figures sit three cells away from their labels. Humans read with their eyes, but the model can only read the string you build for it.

So before any AI is involved, there's a pure pipeline: extract text, normalize encoding, fold layout artifacts into meaningful lines. There's no intelligence in these steps, yet they determine whether the intelligent part can function.

Ollama is an open-source local LLM runtime framework that provides a clean command-line interface and an OpenAI-like REST API, letting developers pull and run open-weight models like Llama, Gemma, and Mistral with a single command. Its key value lies in encapsulating the complexity of model quantization, memory management, and inference backends (CPU, CUDA, Metal), so local deployment no longer requires manually configuring an inference engine. The tool in this article runs a 1B-parameter Gemma on CPU via Ollama precisely because of its ability to automatically handle the GGUF quantization format and thread scheduling, which makes an end-to-end workflow possible with lightweight code at the standard-library level.

PDFs fall into two categories: "digitally native" PDFs with a text layer, and PDFs with only scanned images. The former describe glyph positions internally via content streams; extraction tools can read out characters but often lose reading order. The latter must first go through OCR, with a higher error rate. More troublesome is that in fixed-layout documents, visual tables, stamps, and headers/footers are merely scattered coordinate fragments in the text layer; after extraction you get broken words, scrambled order, and repeated headers. This is exactly why the original emphasizes "folding layout artifacts into meaningful lines"—raw extraction results without layout analysis are nearly unreadable to a small model.

Dividing Again: The Division of Labor Between Deterministic Code and the Model

Once the text is ready, a design principle keeps recurring: everything that must be precise goes to deterministic code; everything that needs to be understood goes to the model.

The code is responsible for finding deadlines, amounts, reference numbers, and issuing institutions. Once these fields go wrong, someone loses money or misses a deadline, so they belong to the part of the system that cannot improvise. The model receives the extracted fields and the cleaned text and does only what it's good at: explaining in calm, everyday Italian what this letter is and what the sender wants.

A 1B model can reliably paraphrase a sentence, but cannot be trusted to read a deadline twice and report it faithfully. This isn't a denigration of the model, but the nature of such models. The pipeline is precisely the way to respect this fact.

"1B parameters" refers to roughly a billion trainable values in the model weights, corresponding roughly to a training scale of billions of tokens. This magnitude belongs to small language models, whose world knowledge and multi-step reasoning abilities are significantly weaker than models of tens of billions of parameters or more, but which can work stably on narrow tasks such as single-sentence rewriting and fixed-format paraphrasing. Quantization (e.g., 4-bit) further compresses weight precision to fit CPU memory, at the cost of reduced reliability on numerically sensitive tasks (such as precisely restating long numbers). This is the root of the original's statement that the model "cannot be trusted to read a deadline twice": small models lack stable attention to numeric tokens, rather than lacking logical ability.

What "Preparing the Document" Actually Means

Specifically, the middle layer does three kinds of work, none of which appears in the demo video:

Cleaning. After extraction you get hyphenated broken words, headers repeated on every page, numbers split across lines. The text must be reassembled into sentences before a small model can do anything reasonable.

Anchoring. Key fields are found by code and explicitly passed to the model, and the explanation is built around values no one has to guess.

Boundary constraints. The model sees only the letter and the extracted fields—no conversation history, no poetic system prompt, no extra context to rely on when the letter is ambiguous. If the letter is unclear, the honest output is "this part is unclear," not a confident fill-in.

The last point deserves the most emphasis. The quality of a small local model is mostly decided before it runs: by what you place in front of it, and by what you deliberately keep away from it.

The context boundary refers to all the input visible to the model in a single inference. Without explicit constraints, system prompts, conversation history, and irrelevant retrieved fragments all get concatenated into the context, and small models are easily distracted by irrelevant content, producing plausible-sounding but unfounded output (hallucination). A common practice is to pass only the cleaned text of the current document and the structured fields extracted by code with each request, keep no cross-letter session state, and explicitly require in the prompt to "state clearly when information is insufficient." This design, trading flexibility for predictability, is the main source of trustworthiness for small models in production environments.

The Only Measurement

On the developer's machine, running on CPU, one measured run: about 40 seconds end-to-end per letter, most of the time spent on the model. The pipeline part takes only milliseconds. If you take away only one thing: the 40 seconds is the boring part. The work that makes the output trustworthy is that middle layer no one screenshots.

The tool genuinely exists and is a personal project; the 40-second figure is a single measurement on a single machine, as the developer has explicitly stated.

Implications for Local AI Practice

The value of this case is that it counterintuitively points out: the bottleneck for small models in practice is often not model capability, but document preprocessing and system design. For developers who want to deploy AI locally to handle sensitive documents, the implications are clear: give precision to code, give understanding to the model, and trade strict context boundaries for reliability. This determines the final experience more than chasing larger models or faster inference.