Gear: Turning RSI into a scikit-learn-Style Framework, Boosting Task Pass Rates by Up to 10 Percentage Points

In research on recursive self-improvement (RSI) systems, a recurring dilemma is that papers keep multiplying, yet they are hard to compare with one another. An open-source project called Gear that recently appeared in the Reddit community tries to change that—its goal is clear: to be "the scikit-learn of RSI."

From an Old Reinforcement Learning Problem to a New RSI Framework

After reading a large number of RSI papers, the project's author found that the field is replaying familiar problems from reinforcement learning and evolutionary search: How do you efficiently find promising modifications (sampling)? How do you confirm that a new variant really improves the agent (evaluation)? How do you avoid premature convergence (diversity)?

What these problems have in common is that they all point to the coupling between search strategy and execution evaluation. Gear's design approach is to separate the two: a task agent produces execution trajectories, and a meta-agent uses those trajectories and evaluation feedback to improve prompts, tools, workflows, and context management. This meta-learning-style architecture allows search decisions to be independent of execution and evaluation, providing a foundation for modular implementations of different RSI methods.

RSI (Recursive Self-Improvement) refers to the loop in which an agentic system uses its own capabilities to improve itself (or its components): improvements produced by the system are redeployed, giving it stronger improvement capability in the next iteration. Unlike one-off model fine-tuning, RSI focuses on whether "improvement capability itself" can be iteratively amplified, so its research core often lies in the design of the search space, reliable evaluation of improvement effects, and avoiding degeneration into local optima during self-reinforcement. In engineering implementations, RSI usually takes the form of a meta-level agent modifying prompts, tool sets, workflows, or context management strategies, while a task-level agent executes specific tasks and returns feedback. Precisely for this reason, RSI naturally inherits and amplifies the exploration–exploitation trade-off and the credit assignment problem from reinforcement learning, which makes cross-paper comparability a prominent pain point in the field.

An Existing Runnable GEPA Variant and Staged Evaluation

Gear has already implemented a working GEPA variant, equipped with shared failure diagnosis and staged evaluation mechanisms. Another consideration for the author is ecosystem compatibility: they hope that methods such as GEPA, RRSI, and Harness² can all be implemented and compared side by side more easily on Gear, rather than each maintaining a separate set of code that is hard to reproduce.

This has practical significance for the reproducibility of RSI research. In the past, evaluation conditions in different papers were often inconsistent, making it difficult for readers to judge whether performance differences come from the methods themselves or from implementation details. If a unified framework can reduce this friction, it will accelerate healthy competition among methods.

GEPA (a Genetic-Pareto-style prompt evolution method) is a class of methods that combines evolutionary search with prompt optimization: it maintains a population of candidate prompts (or program variants), forms generational iterations through sampling, mutation, and evaluation, and uses a Pareto-front-like selection mechanism to retain diverse winning candidates across multiple evaluation dimensions or task instances, thereby alleviating premature convergence under a single metric. Unlike gradient-based optimization, GEPA is a gradient-free black-box search that uses evaluation feedback as its signal, so evaluation cost and evaluation noise directly determine its feasibility. Harness², by contrast, emphasizes iteratively improving the "execution harness" itself—that is, the external orchestration layer around the model, including tool calls, context organization, and workflow structure—extending the object of improvement from model weights or prompts to the entire runtime environment. Methods such as RRSI differ mainly in search operators and evaluation protocols, which are exactly the parts that most need to be aligned when bringing them into a unified framework.

Benchmark Data: The Trade-off Between Task Pass Rate and Cost

The author published a set of task pass rate data under GPT 5.6 Luna with medium reasoning intensity. Between the DSH (Dynamic Search Framework) baseline and the Gear-optimized DSH, AutomationBench Marketing rose from 27.0% to 40.0%, AutomationBench Support rose from 17.0% to 25.0%, and Terminal-Bench 2.1 rose from 52.8% to 62.9%. All three benchmarks showed clear improvements, with Terminal-Bench's absolute gain exceeding 10 percentage points.

At the maximum reasoning intensity on the Marketing benchmark, the optimized harness enabled Luna to reach an 88.88% goal completion rate, higher than Astra + Codex's 84.08%. However, the strict task pass rate was 53% versus 57%, indicating that Luna's advantage is mainly reflected in partial completion rather than strict full-task passing. This is a noteworthy detail: the gap between goal completion rate and strict pass rate reveals that the optimized agent is better at advancing task progress but still lags in the reliability of final delivery.

The cost comparison is even more striking. With both models using the same optimized DSH at maximum reasoning intensity, Luna's estimated API cost averaged $0.0568 per task, versus $1.1858 for Astra—95.21% lower—though this figure excludes optimization and infrastructure costs.

Strict task pass rate and goal completion rate are two different evaluation standards: the former counts success only when all acceptance conditions of a task are satisfied, making it a binary strict judgment; the latter is usually given by a grader or model judge and allows partial completion to earn partial credit, making it more forgiving of behavior that "advances the task but does not fully deliver." When the two show systematic differences, it often indicates that the evaluated system is good at decomposing and executing intermediate steps but has shortcomings in the completeness of final delivery, handling of boundary conditions, or format compliance. Terminal-Bench is a class of agent benchmarks oriented toward terminal/command-line environments, with tasks involving file operations, script writing, and multi-step debugging, emphasizing verifiable results in real execution environments, so its pass rate is relatively sensitive to implementation details and tool stability. AutomationBench, by contrast, divides task sets by business scenario (such as Marketing and Support) and is used to measure an agent's end-to-end completion capability in workflows close to real work.

Evaluation and Diversity Questions That Remain Unanswered

In the post, the author candidly raised two open questions: how to evaluate candidates under a limited budget, and how to maintain diversity. These are precisely the key to whether Gear can succeed as "the scikit-learn of RSI."

The budget constraint on candidate evaluation directly affects the breadth of search—the more expensive evaluation is, the fewer variants can be tried, and the higher the risk of premature convergence. Maintaining diversity involves the exploration strategy of the search space; if the meta-agent always optimizes in the direction of the strongest feedback signal, it may fall into a local optimum. The author does not provide answers to these two questions but openly solicits feedback from the community, which itself reflects that the project is still at an early stage.

In terms of positioning, Gear's value lies not only in a particular set of benchmark numbers but in its attempt to provide a reusable, comparable engineering foundation for RSI research. If it can subsequently attract more methods to plug in and form a consistent evaluation protocol, the impact of such a framework on the field may exceed the performance gains of a single paper.

Candidate evaluation under limited budgets is a classic challenge in evolutionary and search-based optimization: evaluating each candidate requires actually executing the task and scoring it, which is costly, so research often adopts surrogate evaluation, early elimination (successive halving), subset sampling, or noise-aware ranking strategies to distinguish good candidates from bad as reliably as possible with limited samples. A direct risk brought by evaluation noise is "overfitting the validation set"—the search process may select variants that happen to perform better on only a small number of evaluation instances. Maintaining diversity corresponds to population diversity maintenance mechanisms in evolutionary computation, such as binning by behavioral or outcome features (as in the MAP-Elites idea), explicitly penalizing individuals too similar to already selected candidates, or retaining multi-objective trade-off solutions on the Pareto front. Together, these two determine whether the search can escape local optima: the cheaper the evaluation and the stronger the diversity, the larger the effective region the search usually covers, but convergence speed and signal reliability decline accordingly.