[KongchangAI]
· 2 min read· 1,169 words

Five Rigorous Gates Code RL Tasks Must Pass Before Entering the Dataset

Five Rigorous Gates Code RL Tasks Must Pass Before Entering the Dataset

Creeks team shares five admission checks and time-bucketing methods for code agent training tasks.

The article summarizes a code task admission process shared on Reddit by a developer from the data company Creeks, following the Harbor task layout of Terminal-Bench 2. Each task must pass five checks: reference solution passes 5/5, empty solution scores 0/5, cheat probes score 0, 10 reruns produce identical results, and a non-author blind-solves it using only the instructions. After passing, tasks are bucketed by strong-engineer time into 30 minutes to 3 hours, 3 to 15 hours, and long-horizon tiers, with long tasks requiring milestones and partial rewards. The article concludes that preventing reward hacking when tests are visible to the agent remains an open problem—hidden tests prevent cheating but sacrifice the agent's self-checking ability—and that scaling the labor cost of blind solves is an unresolved bottleneck.

Reddit original post screenshot

When building verifiable code tasks for agent training, the real difficulty often isn't writing the task itself — it's deciding when a task counts as "done." A developer from the data company Creeks shared on Reddit the admission criteria their team currently uses: every task must pass all of the following checks before entering the training dataset. The process follows the Harbor task layout used by Terminal-Bench 2, making it quite valuable for anyone doing data engineering for code reinforcement learning.

Five Foundational Gates: From Reference Solutions to Blind Solves

According to the post, every task must pass five checks before entering the dataset, all of which are mandatory. These checks span the full chain from correctness and anti-cheating to stability and comprehensibility.

Two-Way Validation: Reference Solution and Empty Solution

The first check is that the reference solution must pass all 5 out of 5 runs (5/5). The second seems counterintuitive — an empty attempt (the agent changes nothing) must score 0 (0/5). The poster specifically notes that this check "sounds silly," but a test suite that passes on unmodified code actually rewards no valuable behavior. Together these two checks ensure the scoring signal points toward correct solutions while never rewarding inaction.

Anti-Cheat Probes: Closing Obvious Shortcuts

The third check is that cheat probes must score 0. The team actively attempts several obvious cheating tactics: modifying test files, hardcoding expected outputs, deleting failing tests, and special-casing test inputs. If any of these tricks fools the grader, the task is rejected. This is the most direct defense against reward hacking — closing known cheating paths before the task enters the dataset.

Reward hacking is a classic failure mode in reinforcement learning: the agent finds a shortcut that maximizes the scoring signal while violating the task's true intent. In code tasks, common forms include directly modifying or deleting tests, hardcoding return values for test inputs, and catching exceptions to return expected results. The root cause is that the scoring function is only a proxy for the task objective, and the gap between proxy and objective gets exploited by the agent. Countermeasures generally fall into three categories: hiding tests or using held-out test sets, adding behavioral constraints and static checks, and designing randomized tests that are hard to special-case. The cheat probes used by the poster are an active red-team style check — simulating known cheating paths before a task enters the dataset to confirm that the grader rejects those solutions.

Stability and Blind Solves: Eliminating Luck and the Author's Curse of Knowledge

The fourth requirement is that 10 reruns produce identical results, ruling out flaky failures caused by network fluctuations or timing issues. The fifth is what the poster calls the "most expensive" check: someone who did not participate in writing the task must blind-solve it using only the instruction file. If the solver needs to ask the author questions to complete it, the instructions themselves are flawed.

The poster highlights why blind solves are necessary: task authors always know information that never made it into the instructions, and this curse of knowledge can only be exposed through a third-party blind solve.

The curse of knowledge refers to the difficulty experienced by someone who possesses information in imagining the situation of someone who lacks it. In task writing, it manifests as: the author assumes certain environment configurations, implicit conventions, or prerequisite steps are obvious and therefore omits them from the instruction file. An agent or a solver unfamiliar with the background faces the same instructions and fails because that information is missing — and the failure reason has nothing to do with the capability the task is meant to test. A blind solve is the standard way to expose this problem: have a third party who didn't participate in writing the task independently complete it using only the instructions. If they must ask the author questions to make progress, the instructions contain unstated assumptions. The cost of blind solves is mainly labor and typically scales with task difficulty; blind-solving long-horizon tasks is especially expensive.

Time-Based Bucketing: From Thirty Minutes to Long-Horizon Tasks

After passing the above checks, tasks are bucketed by "how long a strong engineer would take to complete them": 30 minutes to 3 hours, 3 to 15 hours, and longer tasks. Tasks exceeding 15 hours go into the long-horizon bucket, which requires milestones and partial rewards. This layered approach acknowledges that tasks of different complexity need different scoring mechanisms — short tasks suit binary pass/fail judgments, while long tasks need fine-grained intermediate rewards to guide the agent.

Partial rewards and milestone mechanisms are core to long-horizon agent task design. When a task takes dozens of hours to complete, the binary success signal is too sparse, and the agent struggles to get useful gradients during exploration, easily falling into ineffective loops. The approach is to break the task into independently verifiable subgoals, granting a small reward for each subgoal achieved while retaining the final overall success reward, and balancing the two through weights so the agent doesn't just farm subgoals without advancing the whole. Milestones also have diagnostic value: they can pinpoint which stage the agent is stuck at. The difficulty is that subgoals must genuinely align with the final objective, otherwise they introduce new reward hacking space — the same dilemma as visible tests.

The Dilemma of Visible Tests: Reward Hacking vs. Self-Checking

The post ends with an open question and the most tension-filled part of the whole process: when tests are visible to the agent, how do you deal with reward hacking? The poster admits that hidden tests help, but then the agent can't check its own work. This is a real trade-off — visibility enhances the agent's ability to self-correct autonomously while opening a window for cheating.

Judging from this gate design, the team's approach is to minimize cheating space at the admission stage (probe checks), then ensure signal quality through stability and blind solves, making the risk of visible tests manageable. But the post doesn't give a final answer for the visible-test scenario — instead it asks the community for practices, showing this remains an open engineering problem.

Implications for Code RL Data Engineering

The value of this process lies in breaking "task quality" into executable, verifiable checks rather than relying on subjective judgment. Two-way validation (reference solution passes, empty solution scores zero) and cheat probes are especially worth borrowing — they're inexpensive yet block most low-quality tasks. Cross-referencing multiple sources, blind-solve checks are often seen in community discussions as the most effective but also most expensive step, and the poster's experience aligns with this consensus.

For teams building training datasets, a practical question is: how do you scale the labor cost of blind solves? The poster doesn't elaborate, but this may be exactly the bottleneck that needs solving next.

This article is based on a single-source Reddit share. The poster is from Creeks, and the process references the task format of Terminal-Bench 2.

#code RL tasks#reward hacking#blind solve#Terminal-Bench 2#agent training data#task admission checks
Share:

Related articles