Three Lessons from Building a Java Repair Tool with Local LLMs

Building AI-assisted development tools doesn't necessarily require cloud-based large models. Developer Njabulo Ncube shared his hands-on experience building a Java repair tool with local LLMs on dev.to, and three of his lessons offer valuable reference for engineers looking to implement similar solutions.

Why Choose Local LLMs Over Cloud APIs

Ncube's project goal was clear: build a tool that can automatically detect and fix Java code issues. Unlike directly calling cloud APIs like GPT-4, he chose to run LLMs locally.

The reasoning behind this decision involves several dimensions. Data privacy is the primary factor—code often contains core enterprise logic, and sending it to third-party servers poses compliance risks. Cost control comes next; if frequent code analysis calls are billed through APIs, the long-term overhead cannot be ignored. Latency is also key—local inference eliminates network round-trip time, making it more friendly for development scenarios that require rapid iteration.

Of course, the local approach has obvious costs: it requires hardware investment, and model capabilities are typically weaker than top-tier cloud models. The key judgment here is that Java code repair tasks are relatively focused and may not need the most powerful general-purpose model.

LLM context window refers to the maximum number of tokens a model can simultaneously receive and process in a single inference, including input prompts and output results. Window sizes vary significantly across models: early open-source models were mostly 2K-4K tokens, while current mainstream local models generally support 8K-32K, with some extensible to 128K or higher. The context window directly limits how much code, dependencies, and documentation the model can "see." In code repair tasks, too small a window prevents the model from obtaining complete type information, leading to incorrect fixes; too large a window may introduce irrelevant noise and significantly increase inference latency and VRAM usage. Common strategies in practical engineering include: prioritizing key dependencies, creating structured summaries of long files, using Retrieval-Augmented Generation (RAG) to inject only relevant fragments, and handling large projects that exceed the window through chunking and iteration.

Lesson One: Context Window Determines Repair Quality

The first point Ncube emphasizes is context management. In Java projects, a class's issues are often tightly related to other dependent classes, interface definitions, and configuration files. If only the erroring code snippet is fed to the LLM, it's difficult for the model to provide a correct fix.

His approach is to include as much relevant type definitions, method signatures, and call chains as possible in the prompt. But local LLMs typically have limited context windows, requiring a trade-off between "providing enough information" and "not exceeding the window." His strategy is to prioritize direct dependencies and summarize indirect ones.

This holds true for any developer building AI programming tools: the quality of context often determines output results more than the model's size itself.

Lesson Two: Prompt Design Should Match How Compilers Think

The second lesson is about prompt engineering. Ncube found that directly describing "please fix this Java code" in natural language doesn't work well. Local LLMs tend to exhibit two types of problems: generated code that is syntactically correct but semantically off, or fixes that address surface issues while introducing new compilation errors.

He switched to a more structured prompt format: explicitly specifying the Java version, imported dependency libraries, expected repair type (type errors, null pointers, deprecated APIs, etc.), and requiring the model to output in a specific format. This "compiler-friendly" prompt design significantly improved repair accuracy.

Essentially, this is using an engineering approach to compensate for local models' capability gaps—since the model isn't smart enough, break the problem down clearly enough.

Structured Prompt refers to breaking down natural language instructions into formats with explicit fields and constraints, such as organizing information using JSON, XML tags, or templated sections. Compared to free text, structured prompts offer advantages: explicit field definitions reduce the model's ambiguous interpretation of intent, fixed output formats facilitate program-side parsing and validation, and reusable templates make debugging and iteration more controllable. In code generation scenarios, common structured elements include target language and version, imported dependencies, input/output examples, constraints (such as prohibiting certain APIs), and expected output format (such as returning only diffs or pure code blocks). OpenAI's Function Calling, Anthropic's Tool Use, and community-popular templates like ReAct and Chain-of-Thought are essentially engineering practices of structured prompts, aiming to use deterministic frameworks to compensate for the model's inherent uncertainty.

Lesson Three: The Verification Step Is Indispensable

Ncube's third lesson may be the most important: never trust code directly output by an LLM.

After the LLM generates a repair solution, his tool automatically calls the Java compiler for verification. If compilation fails, the error information is fed back to the LLM for a second repair, forming an iterative loop. This "generate-verify-regenerate" closed loop is key to ensuring the usability of final output.

This also reveals a broader principle: the LLM's role in code generation tasks should be "proposer" rather than "decision-maker." Final correctness must be guaranteed by deterministic toolchains.

The generate-verify-regenerate loop is an architectural pattern that combines LLM generation capabilities with deterministic tool verification. Its core idea: the LLM handles divergent proposals while deterministic components like compilers, test frameworks, type checkers, or static analysis tools handle convergent judgment. When verification fails, the system feeds specific error information (such as compilation error line numbers, type mismatch details) back to the LLM as new context, triggering the next round of correction. This pattern is widely used in tasks with high correctness requirements such as code generation, mathematical reasoning, and structured data extraction. In engineering, attention must be paid to controlling iteration count (avoiding infinite loops), setting reasonable exit conditions (such as abandoning after two consecutive rounds with no improvement), and maintaining complete logs of each round's input and output for traceability. Similar ideas also appear in academic methods like Self-Refine and Reflexion.

Implications for AI Programming Tool Development

Although Ncube's practice focuses on the specific scenario of Java repair, the principles extracted apply to the broader field of AI-assisted programming. Local LLMs can absolutely deliver practical value in vertical, focused tasks—provided developers are willing to invest sufficient engineering effort in context engineering, prompt design, and result verification.

For teams considering similar solutions, the recommendation is to start from a narrow scenario—such as handling only specific types of compilation errors—get the complete loop working first, then gradually expand coverage. This is far more pragmatic than pursuing "universal repair" from the start.

One-line Summary

Developer builds a Java repair tool with local LLMs, distilling three practical lessons on context, prompts, and verification.

Paragraph Summary

The article shares developer Njabulo Ncube's experience building a Java code auto-repair tool with local LLMs. He chose local over cloud APIs mainly for data privacy, cost control, and inference latency. Three core lessons: first, context window management affects repair quality more than model size, requiring priority injection of direct dependencies; second, adopting structured prompts that specify Java version, dependency libraries, and repair types to compensate for local model capability gaps; third, introducing compiler verification to form a "generate-verify-regenerate" loop, letting the LLM act as proposer while correctness is guaranteed by deterministic toolchains. The author recommends teams start from narrow scenarios, get the loop working first, then expand.