Resolve
A benchmark for evaluating code editing tasks, measuring whether a model completes explicit modification requirements while preserving existing behavior not restated in the prompt. It includes two tasks, resolve and resolve-plus.
Timeline (last 90 days)
The developer Aditya Kumar Puri published an evaluation set named Resolve in the Kaggle community benchmark challenge, specifically measuring the 'half-fix' problem in code editing (completing the required change but losing old behavior that was not restated).
Each Resolve benchmark problem gives a Python function, one modification requirement, and two example inputs; the model can only return the rewritten function, and the hidden tests are divided into two groups, fail-to-pass and pass-to-pass, with the problem counted as solved only if both groups fully pass, and a single test failure zeroes the whole problem.
Resolve's judging logic is borrowed from SWE-bench (Jimenez et al., ICLR 2024), removing scaffolding such as repository fetching and golden patches, keeping only the criterion of 'fail-to-pass plus pass-to-pass all passing'.
Resolve has 16 problems, and the final score is the number of fully solved problems out of 16, ranging from 0 to 1.
A total of 23 models received scores on both the resolve and resolve-plus tasks.
GPT-6.1 Sol and GPT-5.6 Luna are the only models that scored 0.9375 on both tasks, that is, 15 out of 16 problems, and did so in two independent samplings.
A single sample is a very noisy snapshot for models near the passing threshold, and the authors suggest that the next step should be to sample each problem multiple times.
The Resolve benchmark is publicly available on Kaggle, version 1, under the Apache 2.0 license.
All Facts (8)
The developer Aditya Kumar Puri published an evaluation set named Resolve in the Kaggle community benchmark challenge, specifically measuring the 'half-fix' problem in code editing (completing the required change but losing old behavior that was not restated).
50%UnverifiedEach Resolve benchmark problem gives a Python function, one modification requirement, and two example inputs; the model can only return the rewritten function, and the hidden tests are divided into two groups, fail-to-pass and pass-to-pass, with the problem counted as solved only if both groups fully pass, and a single test failure zeroes the whole problem.
50%UnverifiedResolve's judging logic is borrowed from SWE-bench (Jimenez et al., ICLR 2024), removing scaffolding such as repository fetching and golden patches, keeping only the criterion of 'fail-to-pass plus pass-to-pass all passing'.
50%UnverifiedResolve has 16 problems, and the final score is the number of fully solved problems out of 16, ranging from 0 to 1.
50%UnverifiedA total of 23 models received scores on both the resolve and resolve-plus tasks.
50%UnverifiedGPT-6.1 Sol and GPT-5.6 Luna are the only models that scored 0.9375 on both tasks, that is, 15 out of 16 problems, and did so in two independent samplings.
50%UnverifiedA single sample is a very noisy snapshot for models near the passing threshold, and the authors suggest that the next step should be to sample each problem multiple times.
50%UnverifiedThe Resolve benchmark is publicly available on Kaggle, version 1, under the Apache 2.0 license.
50%