SIT-Eval

SIT-Eval (Situation-Based Interactive Turn-by-Turn Evaluation) is a turn-by-turn situational evaluation framework that formalizes Agent behavior as "if situation P occurs, then rule Q must be followed," and drives the evaluation process jointly with an LLM interlocutor and an LLM judge.

SIT-Eval gives up the natural fluency of dialogue in exchange for statistical reliability across scenarios, replacing natural dialogue sampling with adaptive situation sampling.

SIT-Eval's judge receives the complete dialogue history, the current turn's situation, the applicable rules, and the Agent's reply, and outputs a per-rule pass or fail for each rule rather than a vague score.

SIT-Eval orchestrates three LLM components: the interlocutor is responsible for uniformly sampling situations and enacting them; the Agent under test produces replies; and the judge makes rule-by-rule determinations based on the situational rules.

SIT-Eval must maintain four types of state at each turn: the complete dialogue history, the currently activated situation, the set of applicable rules, and the record of rule compliance across turns.

The paper evaluated 8 LLMs over 1,292 dialogue turns and found that models with similar overall scores differ enormously in compliance rate on individual rules.

The weakest areas for all tested models center on ambiguous clarification and Socratic questioning, scenarios that are almost never tested in static benchmarks.

N situations multiplied by M rules would naively enumerate into N×M test cases; SIT-Eval uses uniform sampling to control execution time, but the paper does not give sampling details beyond "uniform."

The speed of round-by-round evaluation is not suitable for running on every commit; a reasonable deployment should be split into three tiers: run the aggregated-metrics fast path on commit, run the full SIT-Eval slow path on release candidates, and continuously sample real conversations in production for offline evaluation.

Adversarial agents may construct replies that deceive the judge; if an agent can infer which situation is being tested, it may target its score-gaming accordingly; if situation definitions are injected with malicious prompts, they may also contaminate the evaluation pipeline.

Means to mitigate the judge's own unreliability include multi-judge cross-checking, blinding of situation labels, pre-validation of situation definitions, and sandbox isolation; production deployment also requires judge confidence scores, manual review of low-confidence cases, and A/B testing of judge versions.

SIT-Eval is suitable for scenarios where agent behavior is highly context-dependent, rare but critical safety and compliance situations need to be tested, and there is willingness to invest in situation taxonomy and rule definitions; it is not suitable for scenarios where conversational naturalness is the primary metric, where domain experts to define rules are lacking, or where the evaluation budget cannot support judge validation.