AI Workflow Acceptance: Using Mandatory Checks to Guard Critical Steps

AI agents can get most of a workflow's configuration right and still leave a broken mandatory approval step. If your evaluation report shows only an average score, that kind of failure is easy to hide behind an impressive number.

OpenAI's October 6 research, Advancing Computer Use with Ironclad, turns this problem into a concrete engineering challenge. Researchers worked with practitioners to define 11 tasks across legal, business, and procurement scenarios, each accompanied by 8 to 50 evaluation criteria.

The report shows GPT-6 Astra in Max reasoning mode achieved an average criteria score of 55.0%, while GPT-5.6 Sol in High reasoning mode reached 41.6%. It's worth noting that these averages describe "how many criteria were satisfied," not "how many workflows were completed perfectly." The reported 19.2-minute versus 37.0-minute per-attempt times are simulated estimates, not measured customer time savings. And these results cover only those 11 research tasks — they do not represent all Ironclad workflows.

One key implementation insight can be drawn from this research: separate the score that measures progress from the conditions that allow a workflow to go live. The design pattern described below is a suggested approach, not a description of Ironclad's actual implementation.

Start with Versioned Requirements

Consider a hypothetical procurement policy: financial approval is required when an amount strictly exceeds $10,000 (excluding tax). Real process owners need to confirm these details — agents should not invent them.

For convenience, amounts are expressed in cents, making the threshold 1,000,000. Then define observable test cases:

• 999,999 cents follows the standard path.
• 1,000,000 cents also follows the standard path, because the rule says "strictly greater than."
• 1,000,001 cents requires financial approval.
• A missing amount should remain "incomplete" rather than silently defaulting to the standard path.
• Editing an existing request so that it crosses the threshold should trigger the review behavior specified by the owner.

The last two cases are supplementary requirements that need to be agreed upon before implementation. They exist to surface ambiguity, not to turn an assumption directly into policy.

Bind these confirmed cases to a specific configuration version. A passing record under yesterday's configuration should not vouch for today's modified one.

Keep Mandatory Checks Independent

Evaluation criteria can measure useful labels, documentation quality, and configuration details. Keep that score for benchmarking and debugging. But separately maintain a set of mandatory check IDs chosen by the process owner.

Here is a Python validation function that operates on observations collected by a trusted test runner. It only accepts an explicit pass with an evidence reference targeting the specified version. Missing, outdated, or ambiguous observations are all treated as "unverified."

def acceptance(required, observations, version):
    if not required:
        raise ValueError("Define mandatory checks before evaluation")

    failed = []
    unverified = []
    for check_id in sorted(required):
        row = observations.get(check_id)
        if (not row or row.get("version") != version
                or not row.get("evidence")):
            unverified.append(check_id)
        elif row.get("status") == "fail":
            failed.append(check_id)
        elif row.get("status") != "pass":
            unverified.append(check_id)

    return {
        "ready": not failed and not unverified,
        "failed": failed,
        "unverified": unverified,
    }

No aggregated score is intentionally introduced into this decision. A high score on optional checks cannot offset a failed mandatory check. Both failed and unverified states block ready from becoming true, while remaining distinguishable in the report.

Here is a set of synthetic test fixtures demonstrating the validation behavior — these are demonstration fixtures, not evidence from a real workflow:

required = {"above_threshold", "missing_amount"}
observations = {
    "above_threshold": {
        "status": "pass",
        "version": "config-7",
        "evidence": "fixture:above-threshold",
    }
}

result = acceptance(required, observations, "config-7")
assert result["ready"] is False
assert result["unverified"] == ["missing_amount"]

An empty mandatory check set raises an error immediately, because a list that accidentally becomes empty should never produce a passing result. Unexpected status values are likewise marked as unverified. This function assumes the input mapping follows the established structure; untrusted inputs should be validated at the boundary before calling it.

This idea of separating mandatory checks from scoring has a precedent in software quality assurance: CI pipelines typically distinguish between blocking tests and informational metrics. The former prevent a merge when they fail; the latter are used only for trend analysis. Applying the same pattern to AI agent acceptance is essentially migrating the concept of a quality gate from code deployment to workflow configuration deployment. The core principle of a quality gate is that passing criteria must be defined before the run and cannot be overridden by the overall run performance — which is exactly why the acceptance() function raises an exception when required is empty rather than returning a pass.

Collect Observations Outside the Agent's Narrative

The validation function above checks references and statuses — it cannot prove that a referenced artifact actually exists or that a result is true. If the same agent can both fabricate observations and rewrite the mandatory check set, this gate provides almost no protection.

The right approach is to have an independent test controller submit known requests in a controlled environment, then read the resulting routing or approval status. Save the request ID, the configuration version under test, the inputs, the expected result, the actual result, and the supporting evidence. Before passing results to the validation function, confirm that the evidence reference resolves to an observation record for that exact case and version.

Use screenshots when visual context matters, but also capture the actual result status. A screenshot of a rule in an editor establishes what a configuration looks like; an actually routed test request establishes something different — what the configuration does for a given input.

Save failures as reproducible cases. "The agent missed some steps" is hard to debug. "Changing a saved request from 900,000 to 1,100,000 cents did not trigger financial review" clearly describes an input change and an observable defect.

The risk identified here is known in security as the self-attestation problem: when a system can both produce the behavior being evaluated and influence the evaluation record itself, the credibility of the evaluation collapses. An independent test controller plays a role analogous to the separation of duties principle in auditing — the roles of submitting test inputs, reading system outputs, and recording evidence must be logically independent from the agent under test. This is especially important for computer use scenarios: the agent can directly manipulate the interface, and therefore could theoretically produce the expected visual state in a screenshot while the underlying routing logic does not actually execute as intended. Relying on actual routing or approval status fields — rather than screenshots alone — is the key measure for sidestepping this risk.

Re-verify After Every Change

After each configuration change, collect fresh observations for every mandatory case. Do not reuse old passing records simply because the visible form looks unchanged. Approval routing may depend on fields, conditions, or downstream edits that screenshots cannot reveal.

For a first pilot, a lean mandatory check suite is more useful than a large collection of vague checks. Add cases incrementally as owners identify new requirements or as failures surface missing edge cases. Continue tracking the broader evaluation criteria so you can see progress that improves things without yet reaching an acceptance threshold.

Also track the human effort required to reach an acceptable result: time spent on preparation, review, remediation, and re-testing. Keep these measurements clearly separate from model execution time and the simulated durations reported in the research.

OpenAI's report explicitly preserves a role for human oversight in complex contract scenarios. The validation mechanism proposed here is not intended to replace that responsibility — it is meant to give owners a clearer view of what passed, what failed, and what no one has verified yet.

Start with one workflow, one agreed-upon version of the rules, and a few meaningful cases. Reliable acceptance decisions begin with a precise definition of what the system must do when it is done.