Fixing Java Projects with Local LLMs: Three Key Lessons

Using a local large language model to automatically fix Java build errors sounds like an elegant closed loop: compilation fails → the model generates a patch → the patch is applied → recompile. But after the developer behind the CompileLoop project, LambdaJedi, implemented this flow, they discovered that the truly difficult part isn't getting the model to write code—it's making sure the model's output can be reliably verified and applied.

CompileLoop is an open-source Java tool that takes a project with a failing Maven build, sends the compilation errors to a local code model, gets a fix suggestion, applies the patch in a temporary copy, and rebuilds to verify. The original source code is never touched—only patches that pass verification are presented to the developer for review.

The most valuable output of this project isn't the tool itself, but the pitfalls the developer hit along the way—they reveal some easily overlooked systemic issues in AI-assisted software development.

The model fixes correctly, but the format can still be wrong

The first problem lies with the unified diff format.

In one benchmark case, the Java compiler reported a missing semicolon. The model accurately identified the location that needed changing and produced the correct fix—adding a semicolon after return a + b. But the problem was that the line counts in the diff header were wrong.

The unified diff header contains information about the number of lines before and after the change, and patch tools rely on these numbers to determine the scope of application. CompileLoop's initial implementation would outright reject a patch when the counts didn't match. This led to an awkward situation: the model gave a completely correct fix, but the tool rejected it because of a formatting detail.

This happened repeatedly. The developer eventually realized that instead of having the model regenerate each time, it was better to recompute the line counts from the diff body, just like git apply --recount. The key point is that this isn't relaxing verification of the actual change—it's using deterministic code to fix a field that can be computed deterministically.

Not every problem needs the AI to retry. Sometimes the better approach is to identify which parts of the output can be safely corrected by ordinary code.

This lesson points to a broader principle: in AI-generated output, some parts require intelligent judgment, while others are mechanical formatting requirements. Handing the latter to deterministic code both improves the success rate and reduces unnecessary model calls.

unified diff is a patch format output by the diff -u command on Unix systems, and it's also the common interchange format for tools like git diff, patch, and git apply. Its core structure is: two lines beginning with ---/+++ identifying the file names before and after the change, followed by several "hunks". Each hunk starts with a header of the form @@ -old_start,old_count +new_start,new_count @@. When applying a hunk, patch tools by default strictly verify both the content at the corresponding position in the old file and the line counts declared in the header; if the counts don't match, patch will error out directly, while git apply provides a --recount option that lets it ignore the header counts and re-infer them solely from the +/-/space prefixes at the start of each line. CompileLoop borrows exactly this idea, delegating purely mechanical, deterministically computable fields like the diff header to code rather than the model, thereby avoiding discarding semantically correct patches due to formatting flaws.

The model saying it's fixed doesn't mean it's actually fixed

The second problem is more subtle.

There was a Java method declared to return int, but it actually returned the string "x". The compiler correctly reported the error. The model's summary said it would change the method's return type to int—but the method declaration was already int. What the model actually modified was the method body, changing return "x" to return a + b.

The patch itself was correct, but the explanatory text was wrong.

On the surface, if the code is fixed, an inaccurate summary seems unimportant. But if the tool's design relies on the model's explanation to make decisions—such as determining which files were modified based on the summary—it will make wrong judgments. CompileLoop's approach is: verify the patch itself, and confirm the result by running the build. The model's summary is only reference context, not evidence of a successful fix.

This design choice reflects a core principle: the AI can propose a solution, but the evidence must come from the actual code and test results. In AI-assisted development tools, distinguishing "what the model claims it did" from "what the model actually did" is a foundational architectural issue.

In AI coding toolchains, models typically produce two types of output: executable artifacts (patches, commands, file contents) and natural-language explanations (summaries, rationales, change descriptions). The explanatory text is produced incidentally by the same generation pass as the model, and it describes what the model "thinks" it did rather than the actual contents of the diff; the two can diverge because of the model misreading the context. A reliable engineering decision chain should be based on the artifacts: first perform static validation on the diff (can it apply cleanly, does it touch protected paths), then obtain an objective signal through a real compile or test run. Explanatory text is suitable as auxiliary context for human reviewers, but should not be treated as a source of truth by tool logic—otherwise it will mislead judgments like "which files were changed" or "whether a new dependency was introduced."

Knowing when to stop asking the AI for help

The third problem involves wasted resources.

Sometimes the model would repeatedly generate exactly the same incorrect patch. The first attempt fails, the tool feeds back the error message, and the model returns a second patch—identical to the first. The third might be the same too. The tool did have a maximum attempt limit and would eventually stop, but each attempt meant a model call and a Maven build.

The developer added two early termination conditions:

• Duplicate patch detection: If the model generates a patch that has already been tried, the loop stops immediately. The patch fingerprint uses Java's String.hashCode(), because the fingerprint is only used for patch comparison within the same process; a collision would only cause an early stop, not allow an unsafe patch to pass.
• Unchanged error detection: If two consecutive attempts fail with the same diagnostic message, it indicates no progress is being made.

Across 60 benchmark runs, these stopping conditions were triggered 27 times, including 21 detections of lack of progress and 6 cases of model provider unavailability. This shows that early stopping isn't a theoretical optimization but a mechanism that frequently takes effect in real runs.

An AI fix loop needs more than just a maximum attempt count—it also needs the ability to recognize whether the next attempt is likely to be useful.

Real-world performance differences across models

The developer tested three local code models on 20 Maven projects containing intentional defects, with three attempts per model per case:

Model	Successful fixes	Fix rate	Average latency
Qwen2.5-Coder 7B	9/20	45%	29s
Qwen2.5-Coder 1.5B	4/20	20%	12s
DeepSeekAd-Coder 6.7B	1/20	5%	39s

Qwen2.5-Coder 7B performed best. But what's really worth noting is DeepSeek-Coder 6.7B's performance—it often correctly identified the fix in its summary, but the generated diff contained context lines that didn't match the source file, causing the patch to be rejected as unsafe to apply.

This result shows that choosing a model isn't just about parameter scale or explanation quality, but also about whether it can output in the precise format the system requires. A model that can "understand" the problem but can't generate an applicable patch has limited value in a real toolchain.

The developer also noticed that Qwen2.5-Coder 7B performed better on test-failure cases than on compilation-failure cases, but with only 5 test samples, this observation needs a larger-scale benchmark to verify.

A code LLM's "format compliance" and "semantic correctness" are two relatively independent capability dimensions. Semantic correctness refers to whether the model understands the cause of the error and provides the correct code change; format compliance refers to whether it can accurately express that change in a format the target tool can consume—for CompileLoop, this means the context lines of the unified diff must match the source file character by character. A context line mismatch causes the patch tool to reject the patch outright, and even if the change itself is completely correct, there's no way to verify it. Parameter scale, the proportion of diff and commit data in the training corpus, and whether the post-training phase aligned on patch formats all affect this dimension. Therefore, when selecting a model, you can't just look at pass rates on things like HumanEval—you also need to do an end-to-end applicability evaluation for the actual output channel (diff, patch, structured JSON, etc.).

Engineering is the real challenge

The most time-consuming part of the CompileLoop project wasn't the AI integration, but the engineering infrastructure around the model:

• A process runner for executing Maven, which needs to handle output buffering to avoid deadlocks, implement timeout control, and terminate the process tree when necessary
• A parser that converts Maven output into structured compilation errors and test failures
• A patch validator that rejects dangerous changes, including path traversal, symlink escapes, and unexpected file operations
• Applying and testing patches in a temporary workspace without modifying the original project
• A fix engine that records attempts, checks fix effectiveness, and detects stalled progress

The project currently has 233 tests, with CI running on Ubuntu and Windows. The test suite uses a mock Ollama server and scripted model responses, so it can run without downloading local models.

The developer clearly documented the current limitations: the temporary copy is not a secure sandbox, build scripts can execute arbitrary commands, and container isolation is an important next step. Regression detection is also an area for improvement—fixing one failing test shouldn't accidentally break another.

"The temporary copy is not a secure sandbox" is a common hidden hazard in tools of this kind. A Maven build will sequentially execute the various plugins declared in pom.xml, including exec-maven-plugin, annotation processors, arbitrary Java code in the test phase, and even direct shell calls via maven-antrun-plugin. In other words, as long as a malicious or tainted patch is accepted, the build process itself is equivalent to executing arbitrary commands on the host. Process-level temporary directory isolation can only prevent contamination of the original source code—it cannot restrict file system access, network requests, or subprocess creation. True isolation requires containers (Docker/Podman), gVisor, or lightweight VMs to constrain the build process, along with its file writes and network egress, within a whitelist. This is also why CompileLoop lists "container isolation" as a next step.

Building AI tools is still engineering

The developer used Claude in the project for architecture discussions and code reviews, but emphasized that using AI to help build a project doesn't absolve you of the responsibility to understand what you're building.

You need to understand why recomputing the diff counts is safe, why the patch itself matters more than the model's summary, and why stopping repeated attempts is better than blindly continuing—these decisions need to be explained and defended, not just made to compile.

The direction CompileLoop explores may be more meaningful than the tool itself: not getting the model to write code, but building systems that can verify the model's work and handle the model's mistakes. The model proposes a fix, the tool checks the proposal, the developer reviews the result—and if the model keeps making mistakes, the tool should know when to stop.

The source code, benchmark cases, and results are open-sourced on GitHub, and the benchmark can be reproduced by pulling the Qwen2.5-Coder 7B model via Ollama.