The Truth About Lagging LangSmith Evals: How Do You Reconcile After CI Passes?

Over in the Reddit LangChain community, someone raised a question that anyone doing serious LLM application engineering will eventually run into: if your LangChain or LangGraph workflow passes in CI and the evaluator gives it a perfect score, but the truly trustworthy result doesn't arrive for days—what does that CI pass actually mean?

The scenario the asker described is a familiar one—human-reviewed document extraction, customer-confirmed answers, generated content validated in downstream systems. What these scenarios share is this: synchronous evaluation can only cover structural and deterministic constraints; it cannot represent final business correctness. A green CI check and the ground truth that comes back days later can completely contradict each other.

The Essence of the Problem: The Temporal Mismatch Between Synchronous Evaluation and Delayed Truth

CI evaluation assumes that "input determines output, and evaluation is available immediately." But in real production, for many tasks the source of ground truth simply doesn't cooperate on that timeline—it comes from humans, customers, or downstream systems, and is inherently delayed.

This creates an awkward state: an evaluation gets marked PASS in CI, but that PASS only holds for "the part that was verifiable at the time"—it makes no promise about truth that hasn't come back yet. If the team treats that PASS as permanent verification, subsequent failures get buried under green records, and regression detection breaks down.

According to the asker, the approach he was considering splits verification into two stages: a synchronous stage responsible for schema validation, deterministic constraint checks, and verification against a versioned evaluation contract, recording case ID, run ID, configuration, and artifact hashes, and marking results requiring external verification as PENDING rather than PASS; and an asynchronous stage responsible for linking delayed-arriving ground truth back to the original run, re-evaluating the recorded output under the pinned contract, distinguishing "confirmed correct," "disagreement," and "missing evidence," and triggering investigation when delayed results contradict the original proxy scores.

The cleverness of this split is that it demotes PASS from a "conclusion" to a "phase status," leaving a hook for later reconciliation.

Ground truth in the context of machine learning and LLM evaluation refers to the label or output that is recognized as the "correct answer" for a given input. In traditional supervised learning, ground truth is usually prepared by an annotation team once before training—cost is controllable, acquisition is immediate. But in LLM application engineering, the source of ground truth has fundamentally changed: it may come from a human reviewer's final judgment, a customer's confirmation of a generated answer, or feedback from a downstream business system after actual execution. These sources are not only slower than the CI stage, but their latency is unstable—it could be hours, it could be days. Delayed ground truth thus becomes its own engineering problem: you cannot obtain it at the moment evaluation occurs, so you can only freeze the run's results and do a "post-hoc alignment" once the truth comes back.

The Hardest Part: Historical Attributability

The asker himself identified the real hard bone—historical attribution. Once the pipeline has been modified, you still need to be able to identify "which contract, which set of inputs, which output, which version of the evaluator was used at the time."

This isn't purely a storage problem—it's a problem of evaluation semantics. If the contract itself can drift, then when you bring old ground truth back days later to reconcile, you may not even be able to say whether "this failure should be attributed to the model or to a contract change." So pinning the contract version, evaluator version, and artifact hash is the precondition for making reconciliation results trustworthy.

This also explains why the asker emphasized that "re-evaluation must happen under the pinned contract"—not evaluating historical outputs with the latest evaluator, but bringing out the exact set of criteria from back then.

An evaluation contract refers to the explicit agreement a team reaches before an evaluation begins about "what inputs to use, what pipeline to run, who judges, and by what standard." It contains at least four types of information: the version of the input dataset or case set, the configuration and artifact hash of the pipeline being evaluated, the version of the evaluator, and the decision threshold or scoring rules. Pinning the contract means these contents are snapshotted at the time of evaluation, and any subsequent reconciliation must reference that snapshot rather than re-running with the latest configuration. Without a pinned contract, when delayed ground truth comes back you're facing an evaluation environment that has already drifted, and the question "is this failure a model regression or a stricter evaluator" becomes unanswerable. This is also why the evaluation contract should have its version information included in CI checks—it simultaneously serves as the anchor for regression detection.

Three Practical Paths

The asker threw several concrete options out to the community, which in practice represent the three approaches teams commonly take:

Update the original evaluation result. Once ground truth arrives, directly overwrite the evaluation record of that original run. The advantage is a single source of truth; the disadvantage is that it breaks the immutability of evaluation history, making it hard to explain during audits which score was produced when.

Create a separate reconciliation run. Leave the original run untouched and create a new reconciliation run to record the delayed ground truth and its matching results. History stays clean and traceable; the cost is that queries require cross-run joins, and reporting logic becomes more complex.

Maintain an external validation ledger. Store a separate reconciliation table outside LangSmith, recording run ID, contract version, ground truth arrival time, and consistency conclusions. This offers the greatest flexibility, but it also means evaluation state is split across two systems.

As for preventing "a single PASS being treated as permanent verification," the community consensus is to make the state explicitly polymorphic: PASS, PENDING, CONFIRMED, DISAGREED, MISSING_EVIDENCE. As long as PENDING is not collapsed into green in reports, regression monitoring won't misjudge.

A validation ledger is a reconciliation record maintained independently outside the evaluation system, typically in the form of a table or database, where each row corresponds to a run requiring external verification. Its fields generally include: run ID, case ID, contract version, evaluator version, artifact hash, ground truth arrival time, and consistency conclusion (CONFIRMED/DISAGREED/MISSING_EVIDENCE). The reason to keep the ledger outside LangSmith is that LangSmith's experiments model assumes evaluation completes when the run ends, and lacks native fields for expressing "ground truth arrives later." The cost of the ledger approach is that state is split across two systems, requiring joins at query time—but it also brings maximum flexibility: you can customize reconciliation cycles, alerting rules, and reporting conventions without being constrained by the evaluation framework's data model.

Implications for LangSmith Users

The value of this discussion isn't just about LangSmith itself—it exposes a structural gap in LLM evaluation engineering: most evaluation frameworks are designed for "instant evaluation," while many real business metrics are inherently delayed.

For teams currently using LangSmith datasets, experiments, and LangGraph, the smallest actionable step is: explicitly introduce delayed ground truth fields and reconciliation states into the evaluation contract, and split cases requiring external verification out of the binary pass/fail judgment. This way CI can still give fast feedback, but the reconciliation stage has a basis to work from, and history remains attributable.

If you're also running into similar temporal mismatches, it's worth first clarifying one thing within your team: when you say PASS, do you mean "verified" or "no problems found so far"? The engineering implications behind these two words differ by an entire reconciliation mechanism.

One-line Summary

Delayed ground truth in LLM applications means a CI pass only represents synchronous verification—requiring two-stage reconciliation and historical attributability.

Paragraph Summary

Starting from a Reddit community question, this article discusses what a CI pass really means when an LLM workflow's evaluation passes in CI but the truly trustworthy result doesn't arrive for days. The core issue is the temporal mismatch between synchronous evaluation and delayed truth: CI can only cover schema and deterministic constraints, not final business correctness. The author introduces the asker's two-stage approach—a synchronous stage doing structural and contract validation and marking externally dependent results as PENDING, and an asynchronous stage linking delayed ground truth back to the original run and re-evaluating under a pinned contract. The article emphasizes that historical attributability is the biggest challenge, requiring pinning the contract, evaluator version, and artifact hash. Finally, it compares three practical paths—updating the original result, creating a separate reconciliation run, and maintaining an external validation ledger—and recommends making evaluation states explicitly polymorphic.