[KongchangAI]
· 2 min read· 1,262 words

Omni-Workspace: A New Benchmark for Evaluating Omni-Modal AI Assistants on Complex Tasks

Omni-Workspace: A New Benchmark for Evaluating Omni-Modal AI Assistants on Complex Tasks

Omni-Workspace benchmarks omni-modal agents on 380 long-horizon multimodal workspace tasks; the best model scores only 65.93.

Omni-Workspace is a complex-task benchmark for omni-modal agents with 380 deep-research tasks across four multimodal workspace types and two difficulty levels, requiring cross-file, cross-modal evidence integration and deliverable production. Across 18 representative models, the best averages only 65.93, exposing weak long-horizon workspace execution. It identifies multimodal context inflation and task-state drift as bottlenecks and proposes OmniFocus, a lightweight method combining working memory management with explicit task-state tracking to improve performance and efficiency.

Omni-Workspace: A New Benchmark for Evaluating Omni-Modal AI Assistants on Complex Tasks

As AI assistants move from "understanding images" to "completing tasks in real multimodal work environments," evaluation standards must evolve accordingly. Most existing omni-modal benchmarks remain at the level of perception and understanding, and they fall short in covering an agent's ability to interact continuously and complete complex tasks within workspaces where heterogeneous files and multimodal information are interwoven. The recently proposed Omni-Workspace benchmark is designed to fill this gap.

Omni-Workspace omni-modal workspace task benchmark

The Next Battleground for Omni-Modal Agents: Complex Workspace Tasks

Omni-modal agents are expected to go beyond multimodal understanding and enter real multimodal environments to complete complex tasks. In such scenarios, information is scattered across heterogeneous files and different modalities, and cross-file, cross-modal dependencies require the agent to continuously coordinate perception, reasoning, and tool use while acquiring new evidence. This "long-horizon" interaction capability is precisely the blind spot of current evaluation systems.

The authors of Omni-Workspace point out that existing omni-modal benchmarks mainly focus on multimodal perception and understanding, with clearly insufficient exploration of completing complex tasks through continuous interaction with multimodal workspaces. In other words, a model being able to "understand" an image or a video does not mean it can shuttle back and forth among a pile of documents, spreadsheets, images, and external retrieval results to ultimately produce a deliverable that meets explicit requirements.

In the evaluation context, there is a key difference between "omni-modal" and the more common "multimodal": multimodal typically only requires a model to process text, images, audio, video, and other inputs simultaneously, whereas omni-modal emphasizes that the model can perceive, reason, and invoke tools to take action across modalities within the same task, with outputs that can be deliverables in any modality. The term "agent" means the system does not passively answer questions but autonomously plans steps, reads files, retrieves information, and iterates execution. Combining the two, the difficulty of omni-modal agents shifts from "breadth of modality coverage" to "consistency and traceability of cross-modal information in long-horizon actions," which also explains why simply scaling model parameters or adding more modality types does not directly translate into higher scores on workspace tasks.

Benchmark Design: 380 Deep-Research-Style Tasks, Four Workspace Types, Two Difficulty Levels

Omni-Workspace contains 380 deep-research-style tasks, covering four multimodal workspace types and divided into two difficulty levels. These tasks require the agent to inspect and process workspace files, retrieve external information when needed, integrate cross-file and cross-modal evidence, and produce output files that meet explicit requirements.

Task design emphasizes "evidence-driven" sequential decision-making: newly collected evidence affects subsequent actions, and the agent cannot merely perform a one-time information read but must dynamically adjust its strategy across multiple rounds of interaction. This closely resembles real knowledge-work scenarios—the workspaces that researchers, analysts, or operations staff face are often in a state where file types are mixed and information references one another.

Evaluation Results: The Strongest Model Averages Only 65.93

The research team evaluated 18 representative models, and the results reveal clear limitations in long-horizon multimodal workspace execution capability. The best-performing model achieved an average score of only 65.93. This score indicates that even among the currently leading models, there remains considerable room for improvement when facing tasks that require continuous coordination of perception, reasoning, and tool use.

Further analysis identified two recurring problem categories: multimodal context inflation and task-state drift. The former refers to the continuous accumulation of multimodal information in the context as interaction progresses, which increases the model's processing burden; the latter refers to the agent gradually deviating from the established task state during long tasks and losing track of the objective. These two problems compound each other in long-horizon tasks and become performance bottlenecks.

"Long-horizon" is the key backdrop for understanding these two failure modes. Short-horizon tasks typically require only one or two tool calls or a single round of reasoning to complete, with limited context growth and easy maintenance of objectives; long-horizon tasks involve dozens or even hundreds of interaction steps, each of which may introduce new images, tables, or retrieval snippets. As the number of steps increases, the context window is continuously filled, the model's attention to early key constraints is diluted, and intermediate conclusions from intermediate steps may be overwritten by subsequent information, causing the agent to "forget" how far it has progressed and what remains. Therefore, long-horizon capability is not a simple extension of short-horizon capability; it imposes additional requirements on memory compression, explicit state representation, and error recovery, which is also why context inflation and state drift often appear together.

OmniFocus: A Lightweight Improvement Approach

Based on these findings, the researchers further propose OmniFocus, a lightweight method that combines multimodal working memory management with explicit task-state tracking, aiming to improve both performance and efficiency. Working memory management addresses the context inflation problem, while explicit task-state tracking targets state drift—the approach directly addresses the two core pain points exposed by the evaluation.

The value of this approach lies in the fact that it does not rely on heavier models or more complex architectures, but instead starts from the agent's two mechanisms of "memory" and "state" to provide a more controllable execution framework for long-horizon multimodal tasks. For teams hoping to deploy omni-modal agents in real products, such lightweight improvements offer more practical reference value.

"Multimodal working memory management" borrows the concept of working memory from cognitive science: under limited capacity, retain only information relevant to the current subgoal and compress completed evidence into summaries or structured records, rather than allowing raw images and long texts to pile up continuously. In agent implementations, common practices include selectively summarizing historical multimodal inputs, clearing context no longer needed according to task stage, and externalizing key evidence into retrievable notes. The accompanying "explicit task-state tracking" writes task objectives, completed steps, pending items, and constraints into an independent state representation that can be read each round, so that the model does not have to infer progress on its own from a lengthy dialogue history. One manages "what to remember," and the other manages "where we are," together forming a controllability safeguard for long-horizon execution.

Significance for the Development of Omni-Modal Agents

The contribution of Omni-Workspace goes beyond a leaderboard. It advances the evaluation focus from "multimodal understanding" to "complex task completion in multimodal workspaces," providing a more realistic yardstick for the evaluation and development of omni-modal agents. As agents are expected to autonomously complete complex tasks, benchmarks like this—covering long-horizon interaction, cross-modal evidence integration, and deliverable production—will become key infrastructure for measuring progress.

For developers and researchers focused on deploying AI agents, this benchmark points to a clear direction: capability improvement does not rely solely on stronger perception models but also requires sustained investment in execution mechanisms such as context management and task-state tracking.

One-line Summary

Omni-Workspace evaluates omni-modal agents on 380 long-horizon multimodal workspace tasks, with the strongest model scoring only 65.93.

Paragraph Summary

Omni-Workspace is a complex-task evaluation benchmark for omni-modal agents, containing 380 deep-research-style tasks across four multimodal workspace types and two difficulty levels. It requires agents to integrate cross-file and cross-modal evidence and produce deliverables that meet specified requirements. The team evaluated 18 representative models, with the best averaging only 65.93, showing widespread deficiencies in long-horizon multimodal workspace execution. The analysis identifies two recurring bottlenecks—multimodal context inflation and task-state drift—and proposes OmniFocus, a lightweight method using multimodal working memory management and explicit task-state tracking to improve performance and efficiency. The benchmark advances evaluation from multimodal understanding to complex task completion, offering a more realistic yardstick for omni-modal agent development.

#Omni-Workspace#omni-modal agents#multimodal benchmark#AI agent evaluation#long-horizon tasks#context inflation#task-state drift#OmniFocus
Share:

Related articles