[KongchangAI]
Company

Creeks

A data company engaged in agent training data engineering. It proposes a five-check process that a verifiable code task must pass before being admitted to a dataset.

Timeline (last 90 days)

Nov 14

A developer at the data company Creeks shared on Reddit five admission-gate checks their team uses to build agent training coding tasks.

Unverified50%
Nov 14

The Creeks team's task admission process references the Harbor task layout used by Terminal-Bench 2.

Unverified50%
Nov 14

Creeks' task admission requires that the reference solution pass all 5 runs (5/5).

Unverified50%
Nov 14

Creeks' task admission requires that an empty attempt (the agent makes no changes) must score 0 (0/5).

Unverified50%
Nov 14

Creeks' task admission requires that cheating probes score 0, with probe techniques including modifying test files, hardcoding expected outputs, deleting failing tests, and special-casing test inputs.

Unverified50%
Nov 14

Creeks' task admission requires that rerunning the task 10 times yield completely identical results, to rule out flaky failures related to network fluctuations or timing.

Unverified50%
Nov 14

Creeks' task admission requires that someone who did not participate in writing the task solve it blind using only the instruction file, which the poster calls the most expensive check.

Unverified50%
Nov 14

Tasks that pass the checks are divided into three buckets by the time a strong engineer would need to complete them: 30 minutes to 3 hours, 3 to 15 hours, and a long-horizon bucket of over 15 hours.

Unverified50%
Nov 14

Long-horizon tasks exceeding 15 hours require setting milestones and using partial reward mechanisms.

Unverified50%
Nov 14

As for how to deal with reward hacking in visible test scenarios, the poster did not give a final answer, indicating that this remains an open engineering problem.

Unverified50%

All Facts (10)

Unverified

A developer at the data company Creeks shared on Reddit five admission-gate checks their team uses to build agent training coding tasks.

50%
Unverified

The Creeks team's task admission process references the Harbor task layout used by Terminal-Bench 2.

50%
Unverified

Creeks' task admission requires that the reference solution pass all 5 runs (5/5).

50%
Unverified

Creeks' task admission requires that an empty attempt (the agent makes no changes) must score 0 (0/5).

50%
Unverified

Creeks' task admission requires that cheating probes score 0, with probe techniques including modifying test files, hardcoding expected outputs, deleting failing tests, and special-casing test inputs.

50%
Unverified

Creeks' task admission requires that rerunning the task 10 times yield completely identical results, to rule out flaky failures related to network fluctuations or timing.

50%
Unverified

Creeks' task admission requires that someone who did not participate in writing the task solve it blind using only the instruction file, which the poster calls the most expensive check.

50%
Unverified

Tasks that pass the checks are divided into three buckets by the time a strong engineer would need to complete them: 30 minutes to 3 hours, 3 to 15 hours, and a long-horizon bucket of over 15 hours.

50%
Unverified

Long-horizon tasks exceeding 15 hours require setting milestones and using partial reward mechanisms.

50%
Unverified

As for how to deal with reward hacking in visible test scenarios, the poster did not give a final answer, indicating that this remains an open engineering problem.

50%

Source Articles