Teaspoon IDE
An IDE that integrates an AI coding agent, supports running local LLMs, and emphasizes context management and repair.
Timeline (last 90 days)
In a pure-CPU environment (Ryzen 5 4500U, 32GB RAM, Ollama), the author benchmarked a local LLM as a coding Agent inside the IDE.
Under the old Agent framework, the qwen3.5:4b model repeatedly read the same file in task T3, claiming 'I read it but can't see the content,' and ultimately exhausted all six rounds and failed the task.
The flaw in the Agent framework is that tool results (file contents, command output) exist in context for only one round after execution, then are silently dropped.
The fixes include: persisting tool results into the run history, and stripping the outer quotes of GREP patterns.
The fix shipped with Teaspoon IDE v1.5.1 under the FSL-1.1-MIT license.
The model repeatedly output `GREP "member.*discount"`, and the literal quotes entered the regular expression, causing zero matches; after the fix, the parser strips the surrounding quotes.
In the control experiment, the same 4B model exhausted all six rounds and failed under the old framework, while after the fix it completed in five rounds (pytest→read→fix→re-verify, correct).
In the control experiment, the 9B model took seven steps including two duplicate reads under the old framework, while after the fix it took three steps with zero duplicate reads, following the shortest path.
In the control experiment, the e4b model completed cleanly in four rounds with zero duplicate reads after the fix.
All raw logs, task definitions and evaluation records have been made public, including the measurement data (labo.mond.jp/articles/localllm-harness-bug.en.html) and the code repositories (github.com/cuculhart/ai-editor-lab, github.com/cuculhart/teaspoon-ide).
All Facts (10)
All raw logs, task definitions and evaluation records have been made public, including the measurement data (labo.mond.jp/articles/localllm-harness-bug.en.html) and the code repositories (github.com/cuculhart/ai-editor-lab, github.com/cuculhart/teaspoon-ide).
50%UnverifiedIn the control experiment, the e4b model completed cleanly in four rounds with zero duplicate reads after the fix.
50%UnverifiedIn a pure-CPU environment (Ryzen 5 4500U, 32GB RAM, Ollama), the author benchmarked a local LLM as a coding Agent inside the IDE.
50%UnverifiedThe flaw in the Agent framework is that tool results (file contents, command output) exist in context for only one round after execution, then are silently dropped.
50%UnverifiedThe fixes include: persisting tool results into the run history, and stripping the outer quotes of GREP patterns.
50%UnverifiedThe model repeatedly output `GREP "member.*discount"`, and the literal quotes entered the regular expression, causing zero matches; after the fix, the parser strips the surrounding quotes.
50%UnverifiedIn the control experiment, the same 4B model exhausted all six rounds and failed under the old framework, while after the fix it completed in five rounds (pytest→read→fix→re-verify, correct).
50%UnverifiedIn the control experiment, the 9B model took seven steps including two duplicate reads under the old framework, while after the fix it took three steps with zero duplicate reads, following the shortest path.
50%UnverifiedUnder the old Agent framework, the qwen3.5:4b model repeatedly read the same file in task T3, claiming 'I read it but can't see the content,' and ultimately exhausted all six rounds and failed the task.
50%UnverifiedThe fix shipped with Teaspoon IDE v1.5.1 under the FSL-1.1-MIT license.
50%