Error Repair Loop
An engineering technique in which a model's generated output is executed in the target environment and the resulting error messages are fed back to the model for correction on retry. It can improve the final usable rate but changes the evaluation protocol.
Timeline (last 90 days)
The error-repair loop is an engineering technique: after the model generates SQL, it is first executed against the target database, and if an error occurs the error message is fed back to the model to regenerate a corrected version, which can be iterated multiple times.
Leaderboards usually use single-attempt (pass@1) as the standard, whereas an error-repair loop amounts to multiple sampling or self-correction; reporting repaired results mixed with single-attempt results overstates model capability and misleads technology selection.
The 64.5% first-attempt accuracy is still far from production-ready, and the error-repair loop remains indispensable in actual deployment.
All Facts (3)
The error-repair loop is an engineering technique: after the model generates SQL, it is first executed against the target database, and if an error occurs the error message is fed back to the model to regenerate a corrected version, which can be iterated multiple times.
50%UnverifiedLeaderboards usually use single-attempt (pass@1) as the standard, whereas an error-repair loop amounts to multiple sampling or self-correction; reporting repaired results mixed with single-attempt results overstates model capability and misleads technology selection.
50%UnverifiedThe 64.5% first-attempt accuracy is still far from production-ready, and the error-repair loop remains indispensable in actual deployment.
50%