Code Edits Pass the Request but Silently Drop Behaviors Never Mentioned in the Prompt

A model correctly edits the line you asked for, but casually deletes an existing behavior you never restated in the prompt. This failure rarely shows up on leaderboards, yet it may be the most dangerous mistake in everyday AI coding. Developer Aditya Kumar Puri released an evaluation set called Resolve in the Kaggle community benchmark challenge, specifically measuring this kind of "half-fix" problem.

What Resolve Measures

Each problem gives a Python function, one explicit modification request, and two example inputs. The model may only return the rewritten function; all other output is discarded. Hidden tests are split into two groups: fail-to-pass verifies that the new behavior works, and pass-to-pass verifies that old behaviors not restated in the prompt are preserved. Only when both groups pass completely is the problem considered solved—timeouts, crashes, missing functions, or empty test suites all count as failures, and one failing test zeroes out the entire problem.

The final score is the number of fully solved problems out of 16, ranging from 0 to 1. The resolve task checks new behavior plus unstated old behavior; resolve-plus additionally appends a set of boundary tests that the prompt never showed. The two tasks are sampled independently, so a model's score on resolve-plus can easily be higher than on resolve.

These rules directly borrow the SWE-bench approach to determining whether an issue is resolved (Jimenez et al., ICLR 2024), removing scaffolding like repository fetching and gold patches, keeping only the hard criterion of "fail-to-pass plus pass-to-pass all passing." The failure mode the author wants to capture is specific: the change accomplishes what was asked, but drops an invariant that the prompt did not re-emphasize.

fail-to-pass / pass-to-pass are commonly used regression verification terms in software testing, systematically applied in code-agent evaluations such as SWE-bench. fail-to-pass refers to test cases that fail before the change and should pass after it, confirming that new functionality or a fix actually works; pass-to-pass refers to test cases that already passed before the change and must continue passing afterward, preventing the change from introducing regressions. The combination means a code change must simultaneously satisfy both "do the required thing correctly" and "don't break existing things"; failure in either group is judged unsolved.

SWE-bench (Jimenez et al., ICLR 2024) is a benchmark that extracts tasks from real GitHub issues and merged PRs in open-source repositories. Models must generate patches from natural-language issue descriptions, then use the repository's existing fail-to-pass and pass-to-pass tests to determine whether the issue is resolved. It relies on real project environments, dependency installation, and test execution, making fetching and running costly. Resolve borrows its decision logic but simplifies the task to a single Python function with preset tests, removing scaffolding such as repository fetching, environment setup, and gold patches, so as to focus at lower cost on the specific failure mode of "whether a change silently drops unstated old behavior."

Which Models Were Tested

A total of 23 models received scores on both tasks. Frontier models include Claude Haiku 4.5, Haiku 5.5, Sonnet 5, Sonnet 5.5, Opus 5.5, GPT-5.4 mini, GPT-5.6 Luna, GPT-5.6 Sol, GPT-6.1 Sol, as well as Gemini 3.5 Flash, Flash-Lite, 3.6 Flash, 3.7 Flash, and 3.8 Flash. Open-source models include Gemma 4 31B, Gemma 4 26B, gpt-oss 20B, gpt-oss 120B, DeepSeek^Ad-R1, GLM-5, Qwen 3 235B, Qwen 3 Coder 480B, and Qwen 3 Next 80B Instruct.

Grok 4.5 and Grok 4.6 are in the Kaggle model list, but the agent returned "model not found," so they could not enter the chart. Qwen 3 Next 80B Thinking timed out three times on the same agent, and the last run only reached problem 3 out of 16, so it was also excluded. Edge cases were honestly labeled rather than quietly filled in with a number.

Key Findings

A Tie at the Top, Followed by a Clear Gap

GPT-6.1 Sol and GPT-5.6 Luna are the only models to score 0.9375 on both tasks—15 out of 16 problems, achieved in two independent samples. The task page rounds this to 0.94.

The next group scores 0.9375 on resolve but drops to 0.875 on resolve-plus: DeepSeek-R1, Claude Sonnet 5.5, and Claude Opus 5.5. They solve 15 problems on the first task and 14 on the second. The lowest overall are Qwen 3 235B and Gemma 4 26B, both at 0.6875 on resolve-plus.

gpt-oss 20B is a counterexample: it goes from 0.75 to 0.875. Because the second task is a fresh sample, the higher resolve-plus score comes from a different generation, not because the boundary test suite helped.

Subtle Differences Among Frontier Models

Opus 5.5 and Sonnet 5.5 both start at 0.9375 and finish at 0.875. GPT-5.6 Sol also starts at 0.9375 but drops to 0.8125, two problems short. Sonnet 5 holds 0.875 on resolve and falls to 0.75 on resolve-plus. Haiku 4.5 and Haiku 5.5 remain at the same 0.875 vs. 0.875—on these 16 functions, the newer small models brought no score improvement. Gemini 3.6 Flash and Gemini 3.5 Flash, absent from the Groq agent, are in the same bracket, while Gemini 3.5 Flash-Lite is at the bottom of the frontier group, at 0.75 on both.

Open-Source Models

DeepSeek-R1 leads the open-source group at 0.9375 and 0.875, matching Opus 5.5 and Sonnet 5.5 in the frontier group. Gemma 4 31B is the only open-source model to hold 0.875 on both tasks. Qwen 3 Coder 480B (0.875 vs. 0.8125) clearly outperforms the larger general-purpose Qwen 3 235B (0.8125 vs. 0.6875), showing that models specifically trained for code have a real advantage on this kind of task. gpt-oss 120B and Qwen 3 Next 80B Instruct remain stable at 0.8125.

Score vs. Price

This analysis did not record token consumption or dollar bills; the x-axis is public output pricing (per million tokens), taken from a price list updated on October 8, 2026. GPT-5.6 Luna is the best-value point: 0.9375 for $1.20 per million output tokens, while GPT-6.1 Sol gets the same score at a $10 unit price. Claude Haiku 5.5 holds 0.875 for $0.50, tying with Haiku 4.5 at $5 and Opus 5.5 at $20.

Here, output price refers to the fee a model charges per million output tokens generated, usually higher than input price because generating output tokens is computationally more expensive. Only output unit prices are listed in the comparison because the cost of a code-editing task is mainly determined by generating the rewritten function. Note that the analysis did not record actual token usage or total billing, so it measures list-price levels rather than the real cost of a specific run; actual cost is also affected by response length, number of retries, and whether reasoning mode is used.

One Average Score Hides Three Things

The first round selected one model from each of four families: Claude Haiku 4.5, Gemma 4 31B, GPT-5.4 mini, and Gemini 3.7 Flash, all solving 14 out of 16 problems. Looking only at resolve, the leaderboard is flat. But looking at the second row, Haiku and Gemma hold 0.875, while GPT-5.4 mini and Gemini 3.7 Flash drop to 0.8125.

These three drops are actually three different failures.

On ring_push, all four models complete the requested change on both tasks but lose an old behavior not restated in the prompt—fail-to-pass passes, pass-to-pass does not. A looser scoring would count these edits as correct; Resolve does not. This mistake reproduces consistently across eight runs.

GPT-5.4 mini's loss on resolve-plus comes from the extra boundary test suite: on truncate, both the requested change and the old behavior pass, but the boundary suite fails. Haiku and Gemma did not lose points this way.

Gemini 3.7 Flash's drop is the result of a second sample on the same set of problems—collapse_name passes the first time and fails the second, and it is not caused by the boundary suite.

In other words, one number hides three situations: a change that breaks an unstated invariant, a change that misses a boundary the prompt never showed, and a different result on a second sample of the same problem. A single generation is enough for collapse_name to go from pass to fail. For models near the passing line, a single sample is a very noisy snapshot, and the author suggests that the next step should be multiple samples per problem.

Benchmark and Citation

Resolve is public on Kaggle, version 1, Apache 2.0 license. The task page lists all run records, including incomplete ones, but only models with both scores completed enter the chart.

• resolve task: https://www.kaggle.com/benchmarks/tasks/puriadityakumar/resolve/1
• resolve-plus task: https://www.kaggle.com/benchmarks/tasks/puriadityakumar/resolve-plus/1

Recommended citation: Aditya Kumar Puri. Resolve. https://www.kaggle.com/benchmarks/puriadityakumar/resolve/versions/1, 2026.

For developers who use AI coding assistants daily, this benchmark offers an important reminder: passing the tests you wrote does not mean you didn't break behaviors you didn't write down. When doing code-editing tasks, in addition to verifying new functionality, you also need to actively check whether existing behaviors that the prompt did not mention, but that the code originally carried, still hold.

One-line Summary

The Resolve benchmark reveals AI's hidden failure mode of correctly making the requested change while dropping prompt-unstated old behaviors.

Paragraph Summary

Resolve is a code-editing benchmark on Kaggle that specifically measures the failure mode in which a model completes the change requested in the prompt but silently breaks existing behavior that the prompt did not restate. Each problem provides a Python function and an explicit modification request, with hidden tests split into fail-to-pass and pass-to-pass groups; only if all pass does the problem score. 23 models were evaluated on both resolve and resolve-plus, and GPT-6.1 Sol and GPT-5.6 Luna were the only models to reach 0.9375 on both, while DeepSeek-R1 led the open-source group with the same score. GPT-5.6 Luna achieved 0.9375 at $1.20 per million output tokens, a notably strong value. The article also notes that resolve-plus score drops may stem from three different causes: breaking an unstated invariant, missing a boundary test, or sampling variance on a second run, making single-sample results noisy.