The Local Receipt Organizer: Gemma + Ollama Build a Privacy-First Open-Source Tool

One of the biggest headaches for freelancers is sorting through invoices scattered across email, photo albums, and screenshots. When tax season arrives, manually entering every receipt becomes an unavoidable burden. What's trickier is that these receipts often contain client names, rates, and payment details—and many people aren't comfortable uploading them to third-party services. Is there a way to enjoy the convenience of automation without sacrificing privacy? Developer DhanushGowdaDev offers one answer: Local Receipt Sorter.

The project runs entirely on the user's own computer. Using the Gemma vision model and the Ollama local inference framework, it performs automatic receipt recognition, field extraction, anomaly detection, and export. It can handle PDFs, photos, and screenshots, extracting vendor names, dates, invoice numbers, currencies, subtotals, taxes, and totals—and it even suggests a standardized filename for each document, such as 2026-03-04_acme-cloud_1180.00.pdf.

Core Design: Local Inference and Structured Output

Local Receipt Sorter's tech stack isn't complex, but every choice revolves around privacy and practicality. The model is the Gemma 3 vision model (4B parameters) served locally via Ollama, so documents never leave the machine. The developer deliberately avoided cloud APIs, so there are no pay-as-you-go bills—which also means predictable costs.

The entire pipeline is driven by Python: PyMuPDF parses PDFs, Streamlit provides the interactive interface, and pandas handles exports. Installation is simple—just three commands: pull the gemma3:4b model, install dependencies, and start the app. If higher accuracy is needed, you can switch to the 12B version via an environment variable.

To make recognition results programmatically processable, the model output is constrained to a JSON schema. No matter what kind of receipt is input, the returned field structure is always consistent. This is crucial for downstream validation and export—otherwise, unstructured text would be difficult to automate.

Ollama is an open-source framework for running large language models locally. It packages model weights, an inference engine, and a REST API into a single executable, so users only need to run ollama pull <model> to fetch and run a model. Its core value lies in abstracting away the complexity of GPU/CPU backend configuration, quantization formats, and VRAM management, enabling non-expert users to run models ranging from 4B to 70B on a personal computer. By default, Ollama exposes an OpenAI-compatible interface at localhost:11434, so Python applications can call local models via standard HTTP without any cloud credentials. For privacy-sensitive tools like Local Receipt Sorter, Ollama's "data never leaves the machine" property is key: prompts, image tensors, and output tokens during inference all stay in local memory and on disk, never passing through any third-party server.

JSON schema-constrained output refers to forcing the model's generated token sequence to conform to a predefined JSON structure, either through prompt engineering or decoder constraints in the inference framework. In Local Receipt Sorter, this means that whether the receipt is a PDF, photo, or screenshot, the model returns consistent field names, types, and nesting levels, such as {vendor: string, date: string, total: number, ...}. This approach sidesteps the traditional LLM parsing problem where output "looks like JSON but actually has missing brackets or extra commas." There are usually two implementation methods: first, writing the schema into the system prompt along with few-shot examples; second, using an inference backend that supports grammar-based decoding (such as llama.cpp's GBNF grammar or Ollama's format: json parameter), which directly masks illegal tokens during sampling. For downstream pandas export and rule validation, a stable schema is a prerequisite for an automated pipeline.

Don't Use Model Self-Assessment—Use Rule Validation

One engineering decision worth noting: the developer did not have the model report its own confidence. The reason is that small models (even vision models) are unreliable at judging their own accuracy. Instead, the system includes a built-in rule checker that automatically flags missing fields, incorrect or future dates, non-positive totals, and rows where subtotal plus tax does not equal the total.

All recognition results are presented in an editable table before export, and users can correct them one by one. This human-in-the-loop approach strikes a balance between automation and accuracy. The model handles the heavy lifting of initial extraction, rules catch obvious errors, and humans provide the final check.

During testing, the author encountered an interesting case. Initially, he tested with an order page (which can generate invoices but is not itself an invoice), and the model returned an empty date column. He initially thought the model had failed and even relaxed the prompt. Later he realized the model was right—there was no date in the document at all. So he adjusted the prompt to explicitly require "never fabricate dates, vendors, or amounts." This small episode also reflects that, with appropriate constraints, local models can faithfully reflect the input rather than forcibly making things up.

Model self-assessed confidence refers to having an LLM attach a score or label indicating "how certain I am" along with its answer. For small-parameter models, this self-assessment is often unreliable: the model may give high confidence to completely wrong extractions, or low scores to correct results. The reason is that confidence calibration requires the model to have accurate metacognition about its own knowledge boundaries, and 4B-level vision models typically lack this ability. Local Receipt Sorter chooses deterministic rules instead of self-assessment, such as checking whether a date is in the future, whether the total is positive, and whether subtotal plus tax equals the total. These rules depend only on mathematical and logical relationships between fields, not on the model's subjective judgment, making them more stable, testable, and auditable. In engineering terms, this shifts "uncertainty detection" from inside the model to external code—a common reliability compensation strategy in local small-model applications.

Testing 25 Documents: Feasibility Validation

The creator tested on 25 documents, including 15 PDFs, 10 photos, and 15 screenshots. It should be noted that this sample size can only prove "it works," not "how reliable it is." The author himself acknowledged this limitation: "I only tested 25 documents; these results show that it's feasible, not how high the success rate is."

A key validation: the app still works after turning off Wi-Fi, confirming that inference truly happens locally. For freelancers concerned about data privacy, this is more convincing than a pure accuracy metric.

The author did not compare the local model with closed-source commercial models, so he does not claim it is "better." His claim is more restrained: a small open-source model, running locally, can accomplish a fairly useful part of this work. This cautious attitude is rare in AI projects.

Gemma 3 vision model (4B) is a lightweight multimodal model released by Google that supports text and image input. The 4B parameter version, after 4-bit quantization, uses about 3-4GB of VRAM and can run smoothly on consumer laptop GPUs or Apple Silicon. Its vision capability lets it directly "see" receipt photos and screenshots without relying on OCR preprocessing. However, at the 4B scale, field extraction accuracy on complex layouts, handwriting, or low-light photos will be noticeably lower than 12B or large cloud models. Local Receipt Sorter allows switching to gemma3:12b via an environment variable, providing an adjustable trade-off between accuracy and resource consumption. For freelancers, if receipts are mostly electronic PDFs, 4B is usually sufficient; if many are phone-captured receipts, upgrading to 12B or adding image preprocessing (such as cropping edges and enhancing contrast) would be safer.

The Dual Possibilities of Open Source and Commercialization

As an entry for the "side hustle" theme in the second week of the Hacktoberfest challenge, Local Receipt Sorter also proposes commercialization ideas: a one-time license sold to freelancers, a monthly subscription, or even a batch-processing concierge service.

Privacy itself is a product differentiator. For users handling sensitive financial information, the advantage of local operation is irreplaceable: no risk of data leakage, no per-receipt billing concerns, and users can freely switch to larger models, tighten prompts, or even fine-tune for specific types (such as GST invoices). The code is open source on GitHub, and next steps include adding test documents, trying larger models, fine-tuning for niche scenarios, and attracting the first paying user.

The value of this project lies in showing a path: open-source small models don't need to pursue generality. As long as they reliably complete a clear task in a vertical scenario, they can build tools with practical value and commercial potential. Privacy is not a gimmick—it's a fact guaranteed by the code structure.

One-line Summary

A developer used the Gemma vision model and Ollama to build a fully local receipt organizer that is privacy-first and open-source for commercial use.

Paragraph Summary

Local Receipt Sorter is an open-source receipt organization tool that runs entirely on the user's computer. Using the Gemma 3 vision model and the Ollama local inference framework, it performs automatic recognition, field extraction, anomaly detection, and export for PDFs, photos, and screenshots. The project deliberately avoids cloud APIs, keeping data entirely on the machine and eliminating the risks of privacy leakage and pay-as-you-go billing. Technically, model output is constrained to a JSON schema, and rule-based validation replaces model self-assessed confidence. Users can correct results in an editable table before export. The author validated feasibility on 25 documents, and the app still works with Wi-Fi off, proving inference actually happens locally. The project also proposes commercialization paths such as one-time licensing, subscriptions, or concierge services, with privacy as the core differentiator.