OptionWorld: A New Benchmark for Diagnosing Failures of LLMs as Long-Term Assistants

Large language models (LLMs) are expected to become long-term personal assistants capable of helping us handle everyday affairs across multiple days. But can they reliably complete tasks that require sustained effort over several days? A recent research paper titled OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants provides a diagnostic framework and reveals key shortcomings in current models' ability to maintain task state.

Why Do Long-Term Assistants Need a Dedicated Benchmark?

Unlike single-turn question answering, a long-term personal assistant must track unfinished tasks and maintain coherence as information and the environment change. The paper points out that merely finding information or initiating an action does not mean the task will ultimately be completed. Existing evaluations often focus only on single-step correctness while ignoring task persistence across time. OptionWorld simulates everyday life tasks and tracks the full chain of information acquisition, judgment, action, and outcome, thereby pinpointing exactly where models fail.

LLM "cutoff time" vs. knowledge cutoff: The "last input before cutoff time" mentioned in the paper does not refer to the model's training data cutoff date, but rather to the last round of environmental feedback or user input the model receives before a failure is determined during task simulation. In long-horizon tasks, the model must continuously receive and process new information from the environment, and the timestamps of this information form part of the task state. Understanding this concept helps distinguish "static knowledge stored in model parameters" from "contextual information dynamically acquired during task execution"—the former is fixed at training time, while the latter is updated in real time during inference. OptionWorld's diagnosis is based on a comparative analysis of how these two types of information are utilized.

Three High-Frequency Failure Modes

In runs containing archived decision inputs, the paper analyzed failure instances. A key finding is that in 35.5% of failure cases, the key facts needed to complete the task had actually already appeared in the model's last input before cutoff time. This means the model didn't "not know"—it simply failed to effectively use known information.

Failure analysis identifies three most common problems:

• Information not acquired or not integrated: The model did not actively seek necessary information, or found it but failed to combine it with the current task goal.
• Actionable opportunities wasted: Even when opportunities to execute actions existed in the environment, the agent did not take action.
• Unnecessary constraints imposed: The agent assumed constraints that the task did not require—for example, believing it could not contact a certain person—thereby hindering task completion.

These failure modes all point to a core deficiency: insufficient ability to maintain task state—including linking facts to unfinished work, continuously following up before deadlines, and distinguishing real constraints from self-imposed assumptions.

Insights from Three Intervention Experiments

The study further designed three follow-up experiments to verify the causes of failure:

1. Supplementing missing facts: In some cases, directly providing the key facts the model was missing could restore task success, indicating that information gaps are a direct cause of some failures.
2. Action reminders: Sending action reminders to the model could restart work, but subsequent steps might still miss deadlines, showing that reminders alone are insufficient to solve time management problems.
3. Correcting erroneous constraints: When the model received a note erroneously prohibiting contact, correcting that note restored the permitted inquiry behavior, confirming that the model blindly accepts externally imposed constraints.

These experiments show that failures are not caused by a single factor but by breaks at multiple points in task state maintenance.

Task State Maintenance: In LLM agent research, task state maintenance refers to the model's ability to continuously track task goals, acquired facts, unfinished sub-goals, and environmental constraints across multiple rounds of interaction or extended time spans. Unlike traditional multi-turn dialogue management, task state maintenance for long-term assistants also involves managing the time dimension (e.g., proactive follow-up before deadlines) and discerning external assumptions (e.g., distinguishing constraints explicitly imposed by the user from those inferred by the model itself). The three failure modes identified in the paper—failure to integrate information, wasted action opportunities, and imposed constraints—can all be viewed as breaks in task state maintenance at different stages. This concept is central to understanding OptionWorld's diagnostic logic and is also a key problem that current LLM agent architectures (such as memory-based agents) attempt to solve.

Implications for AI Assistant Development

The value of OptionWorld lies in the fact that it doesn't just provide a score—it offers fine-grained diagnostic information. For developers, this means models can be improved in targeted ways: enhancing information integration, designing better action-triggering mechanisms, and teaching models to distinguish real task constraints from their own assumptions. For users, it serves as a reminder that current LLMs as long-term assistants cannot yet be fully trusted, especially in tasks requiring multi-day follow-up.

The paper's benchmark and failure analysis point the way for future research: the core challenge of long-term assistants is not single-point capability but maintaining coherent task state. As model scale grows, this challenge may become even more prominent, requiring dedicated architectures and training strategies to address.

One-line Summary

OptionWorld benchmark reveals three major failure modes of LLMs as long-term assistants in task state maintenance.

Paragraph Summary

OptionWorld is a new benchmark for diagnosing why large language models fail as long-term personal assistants. It simulates multi-day everyday life tasks and tracks the full chain of information acquisition, judgment, action, and outcome to pinpoint where models fail. The study finds that in 35.5% of failure cases, the key facts needed to complete the task had already appeared in the model's last input before cutoff time, yet the model failed to use them effectively. Failures fall into three categories: information not acquired or not integrated, actionable opportunities wasted, and unnecessary constraints imposed. Three intervention experiments further show that supplementing missing facts can restore partial success, action reminders can restart work but cannot guarantee deadlines, and correcting erroneous constraints can restore permitted behavior. The core deficiency is the model's insufficient ability to maintain task state.