Shoebox: Build an Offline Receipt-to-Table Tool with Local Gemma 3

When Receipts Pile Up, but the Data Stays Off the Cloud

Freelance designers, small shop owners, contractors—almost every small business owner has a "shoebox" stuffed with crumpled receipts and invoices. At the end of each month, entering them into Excel often eats up an entire weekend. There's no shortage of automation tools on the market, but the vast majority require you to upload your invoice photos to someone else's server. These receipts carry client names, tax IDs, and specific transaction amounts. For a small business, what you're handing over isn't just a few images—it's the lifeblood of your business.

Developer ayushkr23 submitted a project called Shoebox for the second week of the Hacktoberfest Open Source AI Challenge. The idea is straightforward: drag in receipt photos, export a clean table, and keep all data on your laptop the entire time. It runs the Gemma 3 (4B) vision model locally via Ollama, with a pure HTML/CSS/JS frontend and a FastAPI + Pydantic backend for validation. No API key required, and no dependency on any cloud service.

What It Actually Does

Shoebox is a local web app that supports receipt photos in JPG, PNG, WEBP, and HEIC formats. Each image is read by the model, which extracts the date, merchant, currency, subtotal, tax (GST), total, payment method, category, line items, and the model's own confidence score. You can then review, edit, and export to CSV.

What's worth noting is how it handles "uncertainty." When confidence falls below 0.6, a field is missing, or the sum of line items deviates from the total by more than 2%, the corresponding row turns amber. Clicking that row opens a side drawer with the original image on the left and editable fields on the right, making it easy to compare and correct. This design acknowledges the limitations of a small 4B model rather than pretending everything is accurate.

The entire pipeline processes images serially, one at a time. Status updates are pushed via Server-Sent Events (queued, scanning, complete, or needs review), and results are written to SQLite so refreshing the page doesn't lose data. Before inference, images are resized by Pillow to a long edge of 1600 pixels. Exported CSVs include a UTF-8 BOM to ensure the ₹ symbol and dates display correctly in Excel.

Server-Sent Events (SSE) is an HTTP-based server push technology. Unlike WebSocket, SSE is unidirectional: the server can continuously send a stream of text events to the browser without the browser needing to poll. In Shoebox, each receipt is processed serially, with statuses moving from "queued" to "scanning" to "complete" or "needs review." These status changes are pushed to the frontend in real time via SSE, so users see progress without manually refreshing. SSE's advantages are simple implementation, automatic reconnection, and compatibility with ordinary HTTP infrastructure—making it a great fit for this "server processes, client reads status" scenario.

Pydantic is a widely used data validation library in the Python ecosystem. It lets developers define data models (schemas) using Python type annotations and automatically handles type conversion, required-field validation, and error messages. Shoebox uses Pydantic v2 to validate the JSON returned by the model for each receipt, ensuring fields like dates and amounts are correctly formatted. When validation fails, it triggers a repair retry rather than crashing the entire batch—a key element in keeping the pipeline robust.

Why Run Locally Instead of Calling Cloud OCR

The author does an honest accounting in the article. Cloud OCR isn't as expensive as you might think—AWS Textract's receipt/invoice API costs about 1 cent per page. At 300 receipts per month, that's roughly $3. So claiming Shoebox exists to save on API costs would be overmarketing.

The real costs lie elsewhere. First is time: the project assumes 90 seconds of manual entry per receipt (an estimate, not a measurement), which means 300 receipts equals 7.5 hours of mechanical labor per month. Second is privacy: receipts carry client names, tax IDs, and revenue data, and many shop owners don't want to upload them. Third is onboarding friction: a small shop owner won't set up an AWS account and API calls, but a local app that runs with a double-click is a different story.

Because the model runs locally, the marginal cost of the 500th receipt is just a bit of electricity. The author's pricing idea is a one-time license of $49 (₹3,999 in the Indian market), including a year of model and parser updates. With no cloud server bills, charging a subscription felt "dishonest" to him.

Cloud OCR services (like AWS Textract, Google Document AI, Azure Form Recognizer) upload images to the cloud, where the provider's large-scale models perform text recognition and structured extraction. Their advantages are high accuracy, support for tables and forms, and no local compute required, billed per page. Textract's receipt API is about 1 cent per page, which is indeed cheap for low-volume users. But cloud OCR has hidden costs: data leaving your premises creates compliance and privacy risks; you need to configure cloud accounts, API keys, and permissions; and long-term subscription costs accumulate. Shoebox chooses local inference, essentially trading a one-time hardware investment and slightly lower accuracy for data that never leaves home, zero configuration, and zero marginal cost—a good fit for privacy-sensitive small business owners with modest receipt volumes.

Tech Stack: Small Model, Strong Validation, Swappable

Shoebox's architectural choices revolve around one goal: making structured extraction reliable with an open-source small model.

• Model layer: Gemma 3 (4B) receives image inputs through Ollama's /api/chat interface and is constrained to output JSON. The model name is controlled by the MODEL_NAME environment variable—when a stronger Gemma is released, one line change switches it.
• Validation layer: Every model response is validated against a schema using Pydantic v2. Malformed JSON triggers a repair-prompt retry; receipts that still fail are marked "needs review" rather than crashing the whole batch.
• Math check: When line items don't sum to the total, a "Math mismatch" flag is raised.
• Fault tolerance: Corrupted images don't interrupt batch processing, and the persistence layer uses SQLite to ensure edits aren't lost.

The author specifically notes that open models offer more than cost savings: he can inspect the raw multimodal output directly, customize a two-stage repair loop for Gemma 3's JSON behavior, and add a 2% math deviation check without fighting a black-box API. These are things that are hard to do with closed-source APIs.

Ollama is an open-source local LLM runtime that provides a unified CLI and HTTP API, supporting one-click pulling and running of models like Gemma, Llama, and Mistral. It automatically handles model quantization, GPU/CPU scheduling, and memory management, letting developers run vision-language models on a laptop without diving into low-level inference frameworks. Shoebox passes images and prompts through Ollama's /api/chat interface and constrains output to JSON; the model name is controlled by an environment variable, so switching models requires just one config change. Ollama significantly lowers the engineering barrier to "running multimodal models locally"—it's the key infrastructure that makes offline AI tools like this feasible to ship quickly.

Real-World Performance and Honest Boundaries

The author benchmarked on 3 synthetic receipts: a clean café bill, a crumpled hardware store receipt with 3D noise and rotation, and a handwritten freelance taxi voucher. Accuracy was 100% for date, merchant, total, and tax, with an average of 2.32 seconds per receipt on an RTX 4050 laptop GPU (6GB VRAM), Ryzen 7 7840HS, and 16GB RAM.

But the author himself calls this a "smoke test, not a real-world accuracy claim." On heavily crumpled receipts with faded thermal ink, a 4B vision model struggles to tell whether a handwritten number is a tip, a discount, or tax. In one test, a tilted photo had line items totaling ₹430 plus ₹21.50 tax, but the handwritten total was obscured by a crease. Gemma 3 missed the handwritten portion, causing the line items to not match the total. Shoebox's math check caught the deviation exceeding 2%, flagged an amber warning, and guided the user to zoom in on the original image in the side drawer to verify—rather than quietly inventing a balancing number.

The limitations the author lists are equally direct: the test sample is only 4 receipts (3 used for benchmarking), so accuracy is a snapshot, not a guarantee; the 4B model struggles with messy cursive, severe creases, and faded thermal paper; inference speed depends heavily on hardware—2.3 seconds on an RTX 4050 laptop, noticeably slower on a CPU-only machine.

The First Real User and the Business Logic

The author gave Shoebox to a freelance videographer friend to try. The friend dragged in five expense receipts from a shoot day: an equipment rental slip, a fuel receipt, and a crumpled crew lunch receipt. The user reported that he used to end up eating half his crew lunch and taxi costs himself because manual entry was too much hassle—and Shoebox read the merchant and tax amount off a folded bill on the first try.

When asked if he'd pay, the friend said he'd pay a one-time ₹2,500 (about $30) just to avoid a monthly cloud subscription and keep client billing data private. His first feature request was client/project tags to group receipts by shoot and bill clients directly. That request also points to Shoebox's most natural product direction.

A Pragmatic Example of Open-Source AI Tools

Shoebox isn't trying to prove that small local models comprehensively beat cloud services. Its value lies in demonstrating a path many overlook: when the task is vertical enough (receipt-to-table), the output format is structured enough (JSON + schema validation), and there's a human review safety net (amber flags + side drawer), a 4B open-source vision model can produce usable results entirely on-device.

The project is open-sourced under the MIT license. The end-to-end integration tests in eval/test_suite.py cover 8 stages, including static endpoints, health checks, serial Ollama extraction, SQLite inline-edit persistence, corrupted-image tolerance, and BOM-prefixed CSV export—all of which the author says pass. Installation only requires ollama pull gemma3:4b, pip install -r requirements.txt, then running run.bat or run.sh. There's no hosted link, because by design it's meant to run on your own computer.

For small business owners who care about data sovereignty and don't want to pay a monthly subscription for a few hundred receipts, a tool that takes "receipts in, table out, data never leaves home" may be more appealing than a feature-rich SaaS that asks you to upload everything.

One-line Summary

Shoebox uses a local Gemma 3 small model to turn receipt photos into tables, with data never leaving home.

Paragraph Summary

Shoebox is a project by developer ayushkr23 submitted to the Hacktoberfest Open Source AI Challenge, aimed at batch-converting crumpled receipt photos into clean CSV tables while keeping all data local. It runs the Gemma 3 (4B) vision model locally via Ollama, with a pure HTML/CSS/JS frontend and a FastAPI + Pydantic backend for validation—no API key or cloud service needed. The project flags rows with low confidence, missing fields, or amount deviations over 2% in amber, and provides a side drawer with the original image for manual review. The author openly admits cloud OCR isn't expensive (about 1 cent/page); the real selling points are privacy, zero configuration, and zero marginal document cost. On 3 synthetic receipts, the four key fields hit 100% accuracy at an average of 2.32 seconds per receipt, but the author explicitly states this is only a smoke test—the 4B model still makes mistakes on creases, handwriting, and faded thermal paper.