[KongchangAI]
BenchmarkTemporal Action Safety in Conversation

TASC

A benchmark for evaluating the ability of LLMs to apply time-related constraints in dialogue. It contains 3,418 inputs and 33 contrastive scenarios, and is used to disentangle temporal reasoning from temporal application.

Timeline (last 90 days)

Nov 5

The research team proposed the TASC (Temporal Action Safety in Conversation) benchmark to specifically evaluate LLMs' ability to apply time-related constraints in conversation.

Unverified50%
Nov 5

The TASC benchmark contains 3,418 inputs derived from 128 templates.

Unverified50%
Nov 5

In cases where the model's action recommendations actually violated temporal rules, the assistant proactively considered temporal constraints only 30.5% of the time.

Unverified50%
Nov 5

The study tested 13 models and found that even when models correctly computed the elapsed time, they still failed to apply this constraint to conversational recommendations in 62.1% of cases.

Unverified50%
Nov 5

Explicitly providing temporal information in the prompt resolves only about half of the failure cases.

Unverified50%
Nov 5

Models rarely issue temporal warnings without reason, indicating that the low 30.5% compliance rate stems mainly from missed detections rather than false alarms.

Unverified50%
Nov 5

TASC's evaluation framework explicitly decouples temporal reasoning from temporal application and measures them separately.

Unverified50%
Nov 5

The study separately designed a set of 33 scenarios to compare models' ability to compute elapsed time with their ability to apply the computed result to recommendations.

Unverified50%
Nov 5

TASC currently covers template-generated conversational scenarios, while real user conversations are more diverse in complexity and temporal expressions.

Unverified50%
Nov 5

The 62.1% application failure rate was measured under a specific template system, and performance may differ across task domains.

Unverified50%

All Facts (10)

Unverified

The research team proposed the TASC (Temporal Action Safety in Conversation) benchmark to specifically evaluate LLMs' ability to apply time-related constraints in conversation.

50%
Unverified

The TASC benchmark contains 3,418 inputs derived from 128 templates.

50%
Unverified

In cases where the model's action recommendations actually violated temporal rules, the assistant proactively considered temporal constraints only 30.5% of the time.

50%
Unverified

The study tested 13 models and found that even when models correctly computed the elapsed time, they still failed to apply this constraint to conversational recommendations in 62.1% of cases.

50%
Unverified

Explicitly providing temporal information in the prompt resolves only about half of the failure cases.

50%
Unverified

Models rarely issue temporal warnings without reason, indicating that the low 30.5% compliance rate stems mainly from missed detections rather than false alarms.

50%
Unverified

TASC's evaluation framework explicitly decouples temporal reasoning from temporal application and measures them separately.

50%
Unverified

The study separately designed a set of 33 scenarios to compare models' ability to compute elapsed time with their ability to apply the computed result to recommendations.

50%
Unverified

TASC currently covers template-generated conversational scenarios, while real user conversations are more diverse in complexity and temporal expressions.

50%
Unverified

The 62.1% application failure rate was measured under a specific template system, and performance may differ across task domains.

50%

Source Articles