TASC
A benchmark for evaluating the ability of LLMs to apply time-related constraints in dialogue. It contains 3,418 inputs and 33 contrastive scenarios, and is used to disentangle temporal reasoning from temporal application.
Timeline (last 90 days)
The research team proposed the TASC (Temporal Action Safety in Conversation) benchmark to specifically evaluate LLMs' ability to apply time-related constraints in conversation.
The TASC benchmark contains 3,418 inputs derived from 128 templates.
In cases where the model's action recommendations actually violated temporal rules, the assistant proactively considered temporal constraints only 30.5% of the time.
The study tested 13 models and found that even when models correctly computed the elapsed time, they still failed to apply this constraint to conversational recommendations in 62.1% of cases.
Explicitly providing temporal information in the prompt resolves only about half of the failure cases.
Models rarely issue temporal warnings without reason, indicating that the low 30.5% compliance rate stems mainly from missed detections rather than false alarms.
TASC's evaluation framework explicitly decouples temporal reasoning from temporal application and measures them separately.
The study separately designed a set of 33 scenarios to compare models' ability to compute elapsed time with their ability to apply the computed result to recommendations.
TASC currently covers template-generated conversational scenarios, while real user conversations are more diverse in complexity and temporal expressions.
The 62.1% application failure rate was measured under a specific template system, and performance may differ across task domains.
All Facts (10)
The research team proposed the TASC (Temporal Action Safety in Conversation) benchmark to specifically evaluate LLMs' ability to apply time-related constraints in conversation.
50%UnverifiedThe TASC benchmark contains 3,418 inputs derived from 128 templates.
50%UnverifiedIn cases where the model's action recommendations actually violated temporal rules, the assistant proactively considered temporal constraints only 30.5% of the time.
50%UnverifiedThe study tested 13 models and found that even when models correctly computed the elapsed time, they still failed to apply this constraint to conversational recommendations in 62.1% of cases.
50%UnverifiedExplicitly providing temporal information in the prompt resolves only about half of the failure cases.
50%UnverifiedModels rarely issue temporal warnings without reason, indicating that the low 30.5% compliance rate stems mainly from missed detections rather than false alarms.
50%UnverifiedTASC's evaluation framework explicitly decouples temporal reasoning from temporal application and measures them separately.
50%UnverifiedThe study separately designed a set of 33 scenarios to compare models' ability to compute elapsed time with their ability to apply the computed result to recommendations.
50%UnverifiedTASC currently covers template-generated conversational scenarios, while real user conversations are more diverse in complexity and temporal expressions.
50%UnverifiedThe 62.1% application failure rate was measured under a specific template system, and performance may differ across task domains.
50%