When Thumbs Up Fails: How Task-Oriented LLMs Can Close the Loop with Physical Outcomes

The Evaluation Trap in Task-Oriented LLMs: When "Likes" Become Misleading

A developer on Reddit shared a hard-won lesson from building a diagnostic assistant for industrial equipment technicians (chillers, solar inverters, cranes): the standard thumbs up/down mechanism is nearly useless in task-oriented scenarios.

The core problem is the gap between user satisfaction and answer correctness. When an LLM recommends replacing an expensive drive board, a technician might give it a thumbs up because the response was well-formatted and confidently worded — even if the actual fault was just a loose wiring harness, and the machine breaks down again three days later. The user "liked" the answer, but it was objectively wrong.

This reveals a widely overlooked blind spot in evaluation: in domains where real-world outcomes determine correctness, subjective sentiment cannot serve as ground truth. Persuasiveness and factual accuracy are two entirely different curves.

A Four-Step Approach: Replacing User Sentiment with Physical Outcomes

To address this, the team ran a four-week pilot covering 3 technicians and 72 work orders, replacing user ratings with a state machine tied to physical repair outcomes. Here's the full setup they validated — and every data pitfall they hit along the way.

1. The Physical Verdict Loop

Rather than asking for feedback immediately after generating an answer, they trigger a mandatory outcome check when a work order closes a few days later. The technician must record whether the repair "held" or "failed again" in the real world. This binary result is written back into vector memory.

The next time a technician queries the same fault code, the system retrieves historical repair records and calculates a historical success rate directly from the metadata — for example, "this wiring harness fix succeeded 8 out of 8 times in the field." This approach anchors evaluation to observable physical facts rather than language quality.

2. Deduplicating Memory Hits

A vector memory engine extracts multiple semantic facts from a single narrative. During semantic retrieval, one repair story might return 4–5 retrieved facts. If each retrieved fact is counted as a successful repair, the success rate metric gets artificially inflated.

Their solution: deduplicate all memory hits by source document ID before computing any success rate. This is an easy-to-overlook detail that directly impacts statistical reliability.

3. Text Canonicalization

Technicians describe repairs in wildly inconsistent ways. One might write "Re-wire J4 harness" while another writes "Re-terminate J4 power harness with new ferrules." Direct string matching fragments the historical success data for the same fix into dozens of duplicate entries.

The team ran token-overlap matching at work order close time (basic Jaccard similarity) to map free-text recommendations onto a standardized list of known fixes, ensuring outcome statistics actually accumulate against the same entry.

4. Logging the Misses

A counterintuitive finding: logging "this recommendation didn't apply" (e.g., "checked wiring harness, found no defects") is just as important as logging successes.

Without this, even when a machine has a genuine hardware fault, the model will keep recommending popular common-pattern fixes. The absence of negative samples introduces statistical bias, causing the model to confuse high-frequency recommendations with high-accuracy ones.

Why This Approach Changed What the Model Recommends First

The developer's core takeaway: tracking physical outcomes instead of user sentiment fundamentally changed which recommendations the model prioritizes.

The logic here deserves serious thought from any team building task-oriented or hardware-domain LLMs. When the feedback signal comes from real-world success or failure — not a click on a UI — what the model learns is not "what answer sounds professional" but "what solution actually works."

An Extended Engineering Question: Where Do Outcome Metadata Live?

The original post ends with an open question: for LLMs running in task-oriented or hardware domains, how are teams capturing ground-truth validation after the initial model response? Are outcome metadata stored directly in the vector store, or is a separate database used to manage job-level statistics?

This touches on a real architectural choice in retrieval-augmented systems. Vector stores excel at semantic recall, but they aren't naturally suited for structured success-rate aggregation. Mixing the two can introduce the deduplication and aggregation complexity described above, while keeping them separate requires an additional synchronization mechanism.

The core strength of vector stores (Pinecone, Weaviate, pgvector) is efficient approximate nearest-neighbor search, but their data model isn't designed for relational aggregate queries. Mixing structured statistics — success rates, repair counts — with vector metadata avoids cross-system sync overhead, but running queries like "aggregate success rate by fault code" typically requires pulling large numbers of vector entries and summarizing them at the application layer, which is inefficient and hard to keep consistent.

An alternative architecture is to use the vector store exclusively for semantic retrieval, and maintain a separate relational table (PostgreSQL or SQLite both work) storing work order IDs, standardized repair action codes, outcome labels (success/failure), and timestamps — joined to the vector store via work order ID. At retrieval time, you first get candidate document IDs from the vector store, then query the relational store for the corresponding success rate statistics. Responsibilities are clearly separated, and aggregate queries are far more efficient. The tradeoff is maintaining write-sync across both stores at work order close time.

Lessons for Building Reliable Feedback Loops

This case comes from a single developer's experience with a limited sample size (72 work orders), but it carries broadly applicable lessons for task-oriented AI systems:

• Evaluation signals should be grounded in business truth. In domains with objective success/failure criteria, treating subjective satisfaction as an optimization target is dangerous.
• Data hygiene determines metric credibility. Deduplication, text canonicalization, and negative sample logging are unglamorous work — but they directly determine whether your success rate statistics are trustworthy.
• Feedback loops require time delay. Real-world validation often happens days after the response. System architecture must support this kind of asynchronous, delayed result write-back.

For teams pushing LLMs into real production environments, designing a credible ground-truth feedback loop may ultimately matter more than which model you choose.

Background: Key Concepts Explained

Vector Memory Engine is a core component of Retrieval-Augmented Generation (RAG) systems. It works by splitting text into semantic chunks, encoding each as a high-dimensional vector stored in a vector database. At query time, the question is encoded the same way, and the most relevant chunks are retrieved via cosine similarity or similar metrics. The challenge is that a complete repair narrative typically contains multiple independent facts — "the fault symptom was X, I checked Y, and ultimately replaced Z" — which get split into separate vector chunks. When a single query triggers multiple chunk returns, failing to trace them back to the source document effectively inflates one repair event into multiple success records in your statistics. Deduplicating by source document ID is fundamentally about establishing the correct mapping between vector-level "semantic fragments" and business-logic-level "event integrity."

Jaccard similarity is a classic method for measuring the similarity between two sets, calculated as the size of their intersection divided by the size of their union, yielding a result between 0 and 1. Applied to text, sentences are first split into sets of tokens (words or stems), then overlap is calculated. For example, "Re-wire J4 harness" and "Re-terminate J4 power harness with new ferrules" share the tokens "J4" and "harness," giving a Jaccard score of roughly 0.2 — which, combined with a threshold, is enough to classify them as the same repair type. This approach is simple to compute and requires no pre-training, making it well-suited to industrial maintenance domains with relatively fixed vocabulary. By contrast, using semantic vectors for clustering can sometimes assign greater distance to descriptions that differ in wording but are identical in meaning, making the lightweight token-overlap approach more accurate in practice.