DialFork: A New Benchmark for Evaluating Procedural Hallucination in Conversational Agents

Evaluation of conversational AI has long focused on "what was said"—whether facts are correct and knowledge is accurate. But when large language models are deployed as conversational agents, another failure mode has been almost entirely overlooked: every sentence the model says is individually fluent, reasonable, and factually correct, yet it deviates from the conversational procedure it should follow. A research team from IDIAP has named this phenomenon procedural hallucination and released DialFork, the first benchmark designed for evaluating it on raw conversational text.

Beyond Factual Hallucination: The Overlooked "Procedure Drift"

The problem with factual hallucination is straightforward: the model says something wrong. Procedural hallucination is more insidious—the model doesn't get a single sentence wrong, but it jumps straight to a refund when it should be confirming identity, gives a premature conclusion when it should be collecting necessary information, or takes the initiative to advance to the next step when it should be waiting for user confirmation. Individual sentence quality is flawless, yet the overall procedure has derailed.

Such failures are extremely costly in procedure-heavy scenarios like customer service, ticketing, medical consultation, and financial processing. Traditional evaluation uses metrics like semantic similarity and factual consistency to measure outputs, which are precisely insensitive to procedure drift: a derailed conversation may be highly semantically close to a standard one. The research team's core judgment is that semantic similarity cannot substitute for procedural similarity, and evaluation must adopt a different measurement approach.

How DialFork Is Constructed

DialFork's construction approach is "controlled forking." The research team first prepares conformant dialogues that follow the expected procedure, then cuts branches at designated turn positions, injecting four types of hallucination so that the conversation begins to deviate from its intended track from that point on.

The final dataset contains 24 tasks, 12 domains, and 36,278 conversations in total. The entire construction process was validated by 30 human annotators, ensuring that the injected deviations genuinely constitute procedural errors under human judgment, rather than meaningless random perturbations. This design—"using conformant dialogues as the baseline and injecting controlled deviations by turn"—makes the degree and location of deviation quantifiable, rather than relying on post-hoc subjective judgment.

The dataset is open-sourced under the MIT license and hosted at https://github.com/idiap/dialfork, making it easy for the community to plug directly into existing evaluation pipelines.

The Relationship Between Procedural Hallucination and Controlled Forking: The "controlled forking" in DialFork is inspired by the idea of "fault injection" in software testing. Traditional dialogue datasets often only record real interactions between users and agents, which naturally contain a large amount of chitchat, pleasantries, and irrelevant turns, making the boundaries of procedure drift blurry. DialFork, by contrast, first has humans write standard procedures fully compliant with business norms (e.g., "confirm identity → verify order → confirm refund amount → wait for user confirmation → execute refund"), then forces the model at specific turns to perform erroneous actions such as "skipping steps," "premature conclusions," or "bypassing confirmation." This construction makes "deviation starting point," "deviation type," and "deviation length" all independently controllable variables, thereby transforming the originally subjective "this conversation feels off" into reproducible, statistically quantifiable experimental conditions.

Three Granularities: Monitoring, Retrieval, and Real-Time Guardrails

DialFork breaks down procedural deviation evaluation into three tasks at different granularities, corresponding to three types of needs in deployment scenarios:

• Conversation set deviation measurement (monitoring deployed agents): Measures how far an entire batch of conversations deviates from the conformant procedure, used for continuous monitoring of online agents.
• Single conversation deviation measurement (procedural retrieval): Determines how much a single conversation deviates, usable for retrieving similar procedural trajectories.
• First deviation turn localization (real-time guardrails): Identifies at which turn the conversation first derails, which is key to whether real-time guardrails can intervene in time.

The first two tasks impose new requirements on measurement methods: they must not compare content, but rather compare "how the conversation unfolds." To this end, the team proposes two conversation-flow-oriented metrics, FlowKL and FlowScore.

FlowScore: Replacing Semantic Similarity with Procedural Similarity

In conversation-level evaluation, FlowScore significantly outperforms all baseline methods across seven encoders. This result directly supports the team's core argument—semantic similarity cannot substitute for procedural similarity. In other words, even if two conversations are very close in semantic space, the procedural logic they follow may be completely different, and existing metrics are almost insensitive to this.

The significance of these metrics lies in providing a deployable measurement approach: turning "whether the conversation advances in the correct order" into a computable, comparable score, thereby supporting the three practical needs of monitoring, retrieval, and real-time guardrails.

The Technical Intuition Behind FlowKL and FlowScore: FlowKL draws on the idea of Kullback–Leibler divergence (KL divergence), treating a conversation as a "procedural state transition chain" and computing the difference between the distributions of "next action types" across turns in two conversations, thereby penalizing deviations in action order. FlowScore, on the other hand, is a normalized and interpretable score, convenient for directly comparing conversations of different lengths and domains. Neither relies on lexical or semantic embedding similarity; instead, they abstract conversations into "intent–action" sequences and perform sequence-level comparison. The direct benefit of this design is that two conversations with completely different wording but identical procedures receive high scores, while two conversations with highly similar wording but scrambled steps are judged as low-scoring.

Practical Impact on the Industry

DialFork fills a gap in conversational agent evaluation. For teams currently deploying LLMs into procedure-heavy businesses (customer service, appointments, consultations, transaction processing), this means:

Optimizing individual sentence quality cannot solve procedural hallucination. A model may perform excellently at every local step, yet be unreliable overall, and evaluation must explicitly incorporate the procedural dimension.

Guardrails need turn-level judgment capability. Localizing the "first deviation" turn is more practically valuable than post-hoc judging whether an entire conversation is correct, because it determines the timing of intervention.

Evaluation standards need to run in parallel with content evaluation. The team's expectation in their release notes is clear: they hope the community will begin measuring conversational processes in addition to content.

As agents move from "answering questions correctly" to "getting things done," the procedure itself becomes part of product capability. What DialFork provides is precisely the first ruler for measuring this part of the capability.

Background Supplement

Real-Time Guardrails and Turn-Level Evaluation: Real-time guardrails refer to mechanisms that monitor agent output in real time during a conversation and immediately intervene upon detecting risk (e.g., forcing a follow-up question, blocking the current response, or escalating to a human). Such systems are extremely latency-sensitive, so they cannot wait until the entire conversation ends before making a judgment. DialFork lists "first deviation turn localization" as a separate task precisely because guardrails must react at the very turn when the deviation occurs. When evaluating this task, the metric no longer compares the final outcome of the entire conversation, but instead examines whether the model can precisely mark "from here on, the procedure is no longer correct" at the correct turn position. This task is more difficult than post-hoc judgment because as the conversation progresses, deviation signals are diluted by subsequent replies that appear normal.

One-line Summary

IDIAP releases DialFork, the first benchmark for evaluating "procedural hallucination" in conversational agents, replacing semantic similarity with procedural similarity.

Paragraph Summary

The IDIAP research team introduces the concept of "procedural hallucination," referring to conversational agents that are individually fluent and error-free in every sentence yet deviate from the business procedure they should follow, which is extremely costly in procedure-heavy scenarios like customer service and consultation. To this end, they release DialFork, the first benchmark designed for evaluating this on raw conversational text. Through "controlled forking," it injects four types of procedural deviations at designated turns in conformant dialogues, constructing a dataset covering 24 tasks, 12 domains, and 36,278 conversations, validated by 30 human annotators. The benchmark divides evaluation into three granularities: conversation set deviation monitoring, single conversation deviation retrieval, and first deviation turn localization, accompanied by two procedural metrics, FlowKL and FlowScore, which significantly outperform baselines across seven encoders, proving that semantic similarity cannot substitute for procedural similarity.