CLE Benchmark Released: LLM Agents Achieve Only 47.7% on Continual Learning

Continual Learning: The Overlooked Weakness of LLM Agents

Most current evaluations of LLM agents focus on the quality of completing a single task, but rarely ask a more fundamental question: Can an agent learn from experience and continuously improve over long time horizons?

The newly released Continual Learning Exam (CLE) is designed to fill this gap. This large-scale benchmark, built by more than 50 domain experts, specifically measures an agent's ability to "improve through experience" in real professional workflows. It no longer looks at single-point performance, but examines whether an agent can, like a human practitioner, accumulate experience across tasks, correct mistakes, and advance level by level.

Continual Learning is not a new concept in machine learning. It refers to a model's ability to retain old knowledge and use prior experience to improve performance on new tasks when learning from a sequence of tasks arriving in order. The classic challenge is "catastrophic forgetting": when a neural network is fine-tuned on new data, performance on old tasks drops sharply. For LLM agents, continual learning is more engineering-oriented—not updating weights, but accumulating reusable strategies and knowledge in context, external memory, or tool calls so that subsequent tasks benefit. CLE measures exactly this "experience-driven improvement" ability. It intersects with traditional multi-task learning, meta-learning, and lifelong learning, but emphasizes progressive advancement and learning-by-doing in real workflows.

Five-Level Ladder Design: Making Experience Truly "Used"

The core design of CLE is quite ingenious. It contains 120 human-written tasks, each organized into a five-level ladder, totaling 600 levels, covering 12 professional domains and 89 specialized roles.

The key mechanism is progressive unlocking: an agent must pass the current level to enter the next, and subsequent levels build on previously accumulated experience. This means that if an agent cannot extract reusable knowledge from early tasks, it will struggle to go far in later levels. Evaluation is performed by a dedicated evaluator agent based on human-written rubrics to judge each submission.

This design turns "continual learning" from an abstract concept into a measurable engineering metric—not whether it can solve a particular problem, but whether it can get better and better.

Feedback and Revision: Small Gains, but Far from Solving the Root Problem

The study compared two settings: single submission, and providing diagnostic feedback after failure with one allowed revision. Across 17 frontier LLM "framework-model" configurations, feedback and revision brought a median improvement of 5.3 percentage points in completion rate.

The improvement is real but limited. This shows that current frontier models still utilize feedback at a shallow level; diagnostic information helps a little but is not enough to trigger genuine internalization of experience.

LLM Agent refers to a system that uses a large language model as its reasoning core and can autonomously plan steps, call tools, read and write external memory, and adjust behavior based on environmental feedback. Unlike single-turn question answering, agents typically operate in multi-turn "perceive—think—act—observe" loops, and thus have interfaces for accumulating experience: they can write successful strategies into long-term memory, turn failure lessons into checklists, or retrieve historical trajectories in subsequent tasks. CLE's progressive design stress-tests exactly these interfaces—if an agent lacks effective memory writing and retrieval mechanisms, prior experience cannot be invoked in later levels, and the ladder breaks.

Feedback-and-revision is a common interaction paradigm in agent evaluation: the evaluator not only gives pass/fail but also points out error types or improvement directions, allowing the agent to resubmit accordingly. It differs from simple trial-and-error resampling because the feedback carries structured information about the cause of failure. However, whether feedback can translate into capability improvement depends on whether the model can correctly attribute causes and abstract feedback into transferable rules, rather than merely patching the current problem. The median gain of 5.3 percentage points indicates that current models are mostly doing "local patching" and have not yet formed cross-level knowledge accumulation, which is also the key difference between continual learning and single-turn self-correction.

The 47.7% Ceiling: Continual Learning Remains a Hard Nut to Crack

The most telling result is the overall score. The best-performing system passed only 47.7% of levels, and tasks fully completed across all five levels accounted for only 23.3%.

In other words, even the strongest current frontier LLMs cannot pass even half the levels in professional workflows that require long-term experience accumulation, and only a minority can complete the entire sequence. This stands in sharp contrast to single-task benchmarks where scores are often high or even near saturation.

The authors point out that this dataset can be used not only for evaluation but also for post-training to strengthen LLMs' continual learning capabilities. This provides a clear direction for future research: rather than continuing to chase scores on single tasks, make experience learning itself a training objective.

Post-training refers to further training of a model after pretraining, including supervised fine-tuning, reinforcement learning from human or AI feedback (RLHF/RLAIF), and reinforcement learning for specific capabilities. The authors' proposal that CLE can be used for post-training means that "cross-level experience transfer" itself can be set as an optimization objective: for example, training models with curriculum-style task sequences and rewarding reuse of earlier strategies in later levels; or constructing trajectory data that requires reading and writing external memory for fine-tuning. This differs from SFT on static single-task data only; it requires training data to preserve dependencies between tasks so that the model learns to "get better and better" rather than "fight alone."

Implications for Agent Deployment

The value of CLE lies not only in exposing weaknesses but also in redefining the standard for a "good agent." In real professional settings, tasks are often interrelated and experience is transferable. If an agent cannot learn from failure, it will be difficult for it to take on long-cycle work.

For developers, this means that architecture design should place greater emphasis on memory mechanisms, feedback loops, and knowledge accumulation, rather than treating each task as an isolated request. For the evaluation community, CLE also suggests a direction: continual learning ability should become a core evaluation dimension alongside accuracy and reasoning ability.

At present, frontier LLMs have only just begun on this path. The 47.7% level pass rate is both the current state and the starting point to break through in the next stage.