DialFork
A benchmark for evaluating process hallucination in dialogue agents. It constructs conversations that deviate from the prescribed procedure through controlled forking, covering 24 tasks and 12 domains with a total of 36,278 conversations, released under the MIT license.
Timeline (last 90 days)
The IDIAP research team proposed DialFork, the first process hallucination evaluation benchmark for raw dialogue text.
The DialFork dataset contains 24 tasks, 12 domains, and a total of 36,278 dialogues.
The entire construction process of DialFork was validated by 30 human annotators.
DialFork is open-sourced under the MIT license and hosted at https://github.com/idiap/dialfork.
DialFork splits process deviation evaluation into three granularity tasks: dialogue-set deviation measurement (deployment agent monitoring), single-dialogue deviation measurement (flow retrieval), and first deviation turn localization (real-time guardrails).
FlowScore significantly outperformed all baseline methods across seven encoders in dialogue-level evaluation.
DialFork's "controlled forking" construction is inspired by the idea of "fault injection" in software testing.
DialFork's construction method is to cut a branch at a specified turn position in a canonical dialogue and inject four types of hallucination, making the deviation starting point, deviation type, and deviation length independently controllable variables.
Real-time guardrails require turn-level judgment capability; locating the turn of the "first deviation" is more practically valuable than judging after the fact whether the whole dialogue is correct, because it determines the timing of intervention.
All Facts (9)
Real-time guardrails require turn-level judgment capability; locating the turn of the "first deviation" is more practically valuable than judging after the fact whether the whole dialogue is correct, because it determines the timing of intervention.
50%UnverifiedThe DialFork dataset contains 24 tasks, 12 domains, and a total of 36,278 dialogues.
50%UnverifiedThe entire construction process of DialFork was validated by 30 human annotators.
50%UnverifiedDialFork's "controlled forking" construction is inspired by the idea of "fault injection" in software testing.
50%UnverifiedDialFork's construction method is to cut a branch at a specified turn position in a canonical dialogue and inject four types of hallucination, making the deviation starting point, deviation type, and deviation length independently controllable variables.
50%UnverifiedDialFork splits process deviation evaluation into three granularity tasks: dialogue-set deviation measurement (deployment agent monitoring), single-dialogue deviation measurement (flow retrieval), and first deviation turn localization (real-time guardrails).
50%UnverifiedFlowScore significantly outperformed all baseline methods across seven encoders in dialogue-level evaluation.
50%UnverifiedDialFork is open-sourced under the MIT license and hosted at https://github.com/idiap/dialfork.
50%UnverifiedThe IDIAP research team proposed DialFork, the first process hallucination evaluation benchmark for raw dialogue text.
50%