When AI Learns to Say "I'm Not Sure": The Counterintuitive Design of an Allergen Detector

A cake recipe: all-purpose flour, butter, eggs, milk, dark chocolate, cooking oil. Can someone with a soy allergy eat it?

The recipe doesn't contain the word "soy." But dark chocolate usually contains soy lecithin—sometimes it doesn't. Cooking oil in some regions is often soybean oil—sometimes it isn't. The honest answer is neither "yes" nor "no," but rather—go check the packaging.

Developer Abeera started from this seemingly simple scenario and built an open-source tool called Contains. Its core premise is just one thing: when information is insufficient, the model should say "needs verification" rather than offer a confident guess.

Replacing Yes/No Questions with Three-Tier Classification

Contains reads recipes or ingredient lists and outputs a single Contains: statement based on the US nine major allergens (milk, eggs, fish, crustacean shellfish, tree nuts, peanuts, wheat, soy, sesame).

The key design is that the result has three states, not two:

Classification	Meaning
Contains	Guaranteed by the definition of the ingredient itself
Does not contain	No ingredient in the list implies it
Needs verification	A named ingredient may hide the allergen, depending on the brand

The third state is the product. A home baker selling cakes doesn't need a tool to guess for her—she needs to know which three ingredients require checking the packaging and which twelve are safe. Each classification shows the specific words that triggered it and the decision layer that applied, not a black box.

According to the author in the original article, the off-the-shelf model she started with faced 85 hidden allergens and never once said "unsure"—which is exactly the problem she wanted to fix.

The Surprising Baseline Conclusion: Plain Labels Have No Difficulty

Before building a model, Abeera first built a baseline. Zero-shot Qwen3.5-4B achieved 0.98 recall on plain held-out ingredient lists. If the label directly says "milk, wheat flour, soy lecithin," the 4B model can already understand it—there's no worthwhile problem to solve in that direction.

So she constructed a second test set: 62 recipes where allergens hide in compound ingredients and are never named. Dukkah (an Egyptian spice blend), Japanese panko, XO sauce, imitation crab, white sauce—each line hand-written and verified, the file frozen with a recorded hash, not reopened until the very end.

The result redefined the entire project: this was never about "reading labels"—it was about "knowing what the labels don't say."

Fine-Tuning Lost, But Learned a Different Capability

Using the LoRA training API provided by Tinker, the author fine-tuned Qwen3.5-4B on 5,755 records from Open Food Facts. The training set had roughly three parts: 5,000 ordinary products, 550 abstention samples (inserting ingredients from unnamed sources, where only the allergens they might hide become "uncertain"), and 205 contrast samples (same ingredient but with source stated, such as "sunflower lecithin" or "milk chocolate," which should yield a definite answer).

The contrast samples teach the model "unnamed → uncertain, named → classify" rather than being vague about entire food categories.

The result: the fine-tuned model lost to its base model. On the held-out compound recipe test set, the fine-tuned version caught 66%, while the base model caught 78%.

The reason isn't complicated: training labels came from the ingredient text itself, so each sample was essentially "here's a list containing milk, the answer is milk." The knowledge the author actually wanted to teach—that dukkah means sesame, that breadcrumbs mean wheat—wasn't in the labels at all. Like teaching someone to recognize faces using photos where every one has a name tag—all they learn is to read name tags.

But one thing fine-tuning did accomplish: the base model never said "unsure" in 85 opportunities; the fine-tuned version does. The abstention capability was successfully learned.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique: it freezes most of the pre-trained model's weights, injects low-rank adapter matrices alongside certain layers, and updates only these new parameters during training. Compared to full fine-tuning, LoRA uses less GPU memory, trains faster, and can save independent adapters for different tasks without copying the entire model—making it well-suited for rapid iteration on hosted APIs like Tinker.

Abstention means the model explicitly outputs "cannot determine/needs verification" when information is insufficient, rather than forcing an answer. It's an independent evaluation dimension in safety-critical scenarios: a model can have high recall yet never abstain, thereby packaging risky ambiguity as a definite conclusion. That's why the comparison table in this article lists "says unsure" as a separate column alongside recall and confident errors.

The System Matters More Than the Model

What was ultimately delivered is a two-layer structure: a fine-tuned model plus an ingredient rules layer. The merge logic prioritizes safety and is deliberately plain:

• If a rule matches or the model classifies as "contains," output "contains";
• If the model says "uncertain" and the text contains an unnamed source word it can identify, output "needs verification";
• Otherwise output "does not contain";
• Rule-layer matches always override the model's "does not contain."

Full comparison on the frozen test set:

System	Caught	Confident errors	Says unsure	False positives
Keyword baseline	48 (0.56)	37	0	4
LoRA fine-tuned only	56 (0.66)	16	13	0
Qwen3.5-4B zero-shot	66 (0.78)	19	0	4
Delivered system: LoRA + rules	73 (0.86)	11	1	0

Individually, the keyword layer catches only a little over half, and the fine-tuned model underperforms the base model. Together, they exceed every component and are the only configuration with zero false positives. The author's own words: the result isn't the model, it's the system.

Recall is the proportion of all cases that truly contain an allergen that the system successfully identifies—measuring the extent of "missed detections"; the counterpart false positive is the proportion of cases that don't contain it but are classified as containing. For allergen scenarios, missed detections directly relate to personal safety, while false positives create noise and cause people to ignore warnings—so both must be reported together. A "confident error" refers to cases where the system gives a definite conclusion but gets it wrong—a more dangerous failure mode than a simple error.

Failure Cases and a Deliberately Preserved Flaw

Of the 11 confident errors, wheat accounts for 6, milk for 2, eggs for 2, and fish for 1. Specifically: bare "flour" (deliberately excluded by the rules layer, since almond flour and chickpea flour aren't wheat), wheat binders in imitation crab, and white sauce.

White sauce corresponds to the author's favorite bug in the entire project: the rules match bechamel, but the test line wrote béchamel—an accent mark defeated the checker. This issue was discovered after the test set was frozen, so it's recorded as a known finding, not fixed—fixing it would mean modifying the system based on test scores, and the scores would lose their meaning.

There's another design the author says she'd "now change in the opposite direction": that four-line rule that downgrades "uncertain" to "does not contain." Post-scoring review found that 5 of the 11 confident errors were originally correctly classified as "uncertain" by the fine-tuned model, but were overridden by this rule into a definite "no." This traded warning noise for exactly the kind of error the project was meant to prevent. She didn't change it, for the same reason—you can't tune parameters on a test set.

Local Image Reading: Gemma 3 4B's Supporting Role

Manually typing ingredient lists is the most annoying part of this kind of tool. The local version switches to taking photos: Gemma 3 4B runs on the user's own machine via Ollama, reads the label, extracts text, and feeds it into the same pipeline. On a real dairy package, it read the ingredients verbatim and returned Contains: Milk—the rules layer and model layer independently reached the same conclusion, and the evidence tags pointed to the specific matched words.

On CPU, including model loading, it takes about 52 seconds—slow, but perfectly acceptable for an operation done once per product. The OCR text is editable before re-running, because OCR makes mistakes and the user is the one holding the package.

Ollama is a local large model runtime that handles model downloads, quantized loading, and inference service wrapping, letting models run offline on the user's own machine. The significance of local deployment is that ingredient photos and recognition results never leave the device—privacy stays intact; the cost is that CPU compute limits make cold starts and single inferences slower. Gemma 3 4B serves here as a vision-language model (VLM), converting label images to text and passing them to the same classification pipeline as the online version, so both entry points share the same logic.

Engineering Details: Cold Starts and Degradation

The online demo is deployed on Render's free tier, where instances go to sleep. When the author tested her own live site, she hit the worst case: timeout set to 25 seconds, warm-up requires 34 seconds—the first request after any quiet period silently falls back to pure rules mode—and that request is most likely from a reviewer or customer.

The fix wasn't to shorten the timeout, but the opposite: on cold start, the app now waits, tells the user the model is waking up, then returns the real answer. "Quiet degradation is worse than loud slowness." During the fix, a second bug was discovered: the label endpoint blocked the event loop during the wait, which would freeze health checks under concurrency.

Feedback from the First User

The author gave the tool to a home baker who sells cakes, testing it with her three recipes. The expected takeaway was whether the tool was accurate; what she actually learned was what she was competing against.

Her current allergen-checking method is Googling it or asking ChatGPT—a chatbot that answers confidently, never says "unsure," and can't point to which word in the recipe triggered the judgment. This benchmark was more sobering to the author than any evaluation number.

Her conclusions from testing three recipes: the confirmed allergen list had no errors; the chocolate-soy association, and the composition of pistachio paste and banaspati (a South Asian shortening), were new to her; the uncertainty prompts were useful, but only when precise—if every chocolate triggers an alert, it becomes ignored noise. The weakest result she pointed out was the peanut flag on unspecified cooking oil, which the author acknowledged was the system's most strained inference.

Asked whether she'd pay: she'd use it, but it's not time yet—two or three dollars a month, provided it's reliable enough for preparing products for customers and can remember her frequently used brands. The author considers this the most valuable sentence of the entire week: not "great," but "it's not worth the money yet, and here's the specific gap that would make it worth it."

What Open Source Means Here

The author gives three reasons, all from the actual experience of this week.

First, what needs to be inspected is exactly what needs to be fixed. After fine-tuning failed, the only way to figure out why was to be able to read her own training data, labels, and merge rules. If the model were an opaque API, "it didn't work" would be the end of the investigation rather than the midpoint.

Second, the data is open, and so are the flaws. Open Food Facts is a crowdsourced database containing HTML markup, inconsistent allergen labels, and products with self-contradictory claims in their ingredient lists. These problems are visible. A closed dataset with the same problems would produce the same inflated numbers—and no one could tell.

Third, the beneficiaries can't afford enterprise prices. A user running a cake business from her home in Pakistan explicitly said she'd pay two to three dollars a month. A 4B open-source model fine-tuned for $2.50 plus free-tier hosting can serve her perfectly; the pure local version is completely free.

Scope Statement

The author lists the limitations plainly: the data is predominantly US and European, a real mismatch for a Pakistani baker—banaspati appears in the rules because she manually added it, not because the data taught the model; fish and crustacean samples total only 799 positive examples; the gold standard set has only 100 rows, all hand-verified by her alone.

The most important one: abstention behavior is fragile to wording. Sampling the same input 8 times yields 8 identical outputs, but reordering the same ingredients flips soy and peanuts from "uncertain" to a confident "no." So she says she can't claim this capability is reliable—and therefore doesn't.

Every screen in the tool says: this is not a substitute for manufacturer labels.

The Real Value of This Project

Why is a project worth reading when fine-tuning failed, accuracy is 86%, and it still confidently misses allergens in a few cases?

Because its value isn't in the scores—it's in demonstrating a way of building AI products: first construct a test set that can kill your idea, then decide whether to build it; write failures and bugs into the report by name; freeze the test set, and when flaws are discovered afterward, leave them rather than fix them, because fixing pollutes the evidence.

The author writes at the end: what she originally wanted to build was a model that knows what's hiding in your chocolate. What she ultimately delivered is a model that knows when to let you go look for yourself.

For someone responsible for a customer's allergic reaction, the honesty of the latter may be more useful than the confidence of the former.

Background Notes

Zero-shot means letting the model complete reasoning directly from instructions alone, without providing any task examples; the contrasting few-shot approach includes several input-output examples in the prompt. The purpose of a zero-shot baseline is to establish the lower bound of "what the model can already do," thereby determining whether a direction truly has a worthwhile problem to solve.

A held-out set is data that doesn't participate in any training or parameter tuning, used only for final evaluation. This article goes further by freezing the test set after construction and recording its hash, ensuring no one modifies it during evaluation and avoiding results being contaminated by after-the-fact adjustments.

One-line Summary

Contains uses three-tier classification—"contains/does not contain/needs verification"—to teach allergen models to honestly say they're unsure.

Paragraph Summary

The open-source tool Contains classifies recipes and ingredient lists against the US nine major allergens, designing results as three states: "contains, does not contain, needs verification," with "needs verification" being the core of the product. The author first used a zero-shot model to achieve 0.98 recall on plain labels, proving that reading labels isn't the difficulty; she then constructed a test set of 62 compound recipes with hidden allergens, revealing that the real task is "knowing what the labels don't say." LoRA fine-tuning actually underperformed the base model on hidden allergens (66% vs. 78%), yet taught the model to abstain for the first time. The final delivered two-layer system of fine-tuning plus rules achieves 86% recall with zero false positives, far exceeding any single component. The author deliberately preserves two known flaws to maintain the evidentiary integrity of the frozen test set.