Self-Healing Agent Loops: Using AI to Automatically Discover and Fix Programming Language Bugs

When "Software Factory" Moves from Concept to Reality: Boundary's Experiment

"Software Factory" has become a buzzword in AI engineering circles lately, but it's often portrayed as a massive, complex, and intimidating system. The team at Boundary sees it differently — it's really just a bunch of loops.

A software factory is a system architecture where AI Agents serve as the core execution units, linking together various stages of software development (requirements analysis, code generation, testing, deployment, and feedback collection) into an automated pipeline. While the concept dates back to Japanese software engineering practices in the 1980s, it has taken on a whole new meaning with the rise of large language models. The core challenge isn't any single Agent's capability — it's designing collaboration protocols between Agents, translating unstructured human intent into Agent-executable tasks, and identifying the optimal points in the automated workflow where human judgment should intervene.

On the AI the Work show, Boundary co-founder Vaibhav and summer intern Dylan shared an internal system they built: an AI Agent-driven continuous improvement loop for their programming language, BAML. The core idea is refreshingly simple: "We just want the Agent to keep writing BAML code and see what breaks."

BAML (Basically A Made-up Language) is a domain-specific language (DSL) developed by Boundary, designed specifically for AI function calling. It lets developers define LLM input/output structures in a type-safe way, and automatically handles prompt templates, output parsing, and retry logic. Compared to calling the OpenAI SDK directly, BAML decouples "how to interact with an LLM" from business logic, making AI calls testable and maintainable — like ordinary functions. The BAML compiler translates these declarative definitions into client code for multiple languages (Python, TypeScript, Ruby, etc.), so the correctness of its type system has a significant downstream impact on users.

As host Dex summarized: the key to a software factory is "how you take signals from the world — whether it's a CEO's vision, a PM's decision, or a production error — and turn them into work that Agents can execute, while giving them the ability to self-check and self-direct."

Why Does a Programming Language Need to "Self-Heal"?

Building a programming language is hard — not making it work, but keeping it stable over time. Vaibhav put his finger on the fundamental tension:

"The problem with programming languages is that as you keep adding new features, those features compete with each other. Every time you add a new competing feature, it becomes nearly impossible to build tests that cover the entire system."

This is exactly why Python still has bugs after more than two decades, and why Linux does too. Once you have a sufficiently complex core system, the interactions between components make "zero bugs" an almost unachievable goal.

A Concrete Example: Type Inference for the Retry Function

Vaibhav demonstrated a language feature that looks simple but is surprisingly tricky. Type inference is the mechanism in modern programming languages that lets the compiler automatically deduce the type of a variable or expression. It was first systematized in the ML language family in the 1970s through the Hindley-Milner type system. However, when a language introduces higher-order functions, generics, and exception propagation, the computational complexity of type inference rises sharply — in some cases it becomes undecidable.

Imagine a foo function that throws a string. When you wrap it with retry, which accepts a lambda, the resulting demo function should throw whatever type foo throws. If foo changes from throwing a string to throwing an int, the entire type inference chain must update accordingly. This is essentially the problem of "polymorphic propagation of exception types": when the exception signature of a wrapped function changes, all type annotations up the call chain need to update in sync. Similar design trade-offs appear in Rust's ? operator, Java's checked exceptions, and Haskell's monad transformers.

And that's just one feature. When you stack on destructors, defer, multithreading, and more, they start conflicting with each other. The team eventually realized: no matter how well-crafted the tests or how much headcount you throw at it, the traditional approach simply doesn't scale.

The Core Architecture of the Self-Healing Loop

The system Dylan designed is essentially two nested closed loops, where AI Agents collaborate to automate the "discover a problem → fix a problem" cycle.

Loop One: From Code Attempts to Generated Issues

The system dispatches Agents to write large amounts of BAML code — from competitive programming problems to "re-implement this open-source GitHub repo entirely in BAML." As the Agents work, they log every problem they hit, every crash, every place where something doesn't work.

There's a clever design here — the distinction between cold start and warm start:

• Warm start: The Agent is given BAML skill documentation upfront, with tools and context pre-loaded, enabling it to complete tasks more efficiently.
• Cold start: The Agent is forced to find the skill documentation from scratch, simulating the experience of a real user's first time.

A cold start is essentially an automated "new user journey test" — it captures friction points that experienced users would habitually skip over and never discover. For example, the system once found that an Agent couldn't locate the command to install a skill. That itself is a usability bug worth fixing.

A separate Agent then analyzes the complete chat log and generates what the team jokingly calls a "trophy" report. This report doesn't just find bugs — more importantly, it understands what's working and what isn't, identifying genuine positive feedback loops. Findings are ultimately split into two categories: Issues (bugs in the language itself) and Inefficiencies (problems related to the skill documentation).

Deduplication: Avoiding Reporting the Same Problem Twice

How do you prevent the system from repeatedly surfacing the same issue? The team built a deduplication (dedupe) mechanism using multiple prompts. Each Issue comes with "evidence" — the specific chat log that triggered the bug. When the same issue is found multiple times, new logs are appended to the existing Issue as additional evidence.

This created an unexpected bonus: data-driven language design. For instance, an Agent repeatedly tried to write string.split and failed, in two separate tasks: "reversing words" and "Huffman encoding." This signaled that users genuinely need a simple method for splitting on whitespace — a need that surfaced naturally from the data, rather than from guesswork.

Human-in-the-Loop Leverage Points

The system isn't fully automated — it's carefully designed with leverage points for human intervention. This is one of the most valuable insights from the whole discussion.

All Issues exist as Linear tickets, with a label system representing different stages in the pipeline. When an Agent-generated Issue isn't good enough, a human can add a comment, set the label to "redraft," and trigger a redraft Agent to rewrite it based on that comment. Vaibhav demonstrated this live: he reclassified a ticket that had been mislabeled as a "skill issue" as a "BAML describe issue," and the Agent immediately rewrote the entire ticket.

"My goal is that I have a bunch of other things to do. Now as an engineer, I have a background task that's just slowly pushing this pipeline toward completion. I'm not fully engaged — I'm semi-engaged — but what I'm really measuring is quality."

This "semi-engaged" mode of working is the essence of the software factory — engineers shift from being code writers to quality gatekeepers and direction setters.

Why Linear Is Essential

Linear is a project management tool for engineering teams, known for its smooth UI and keyboard-first interaction design. Dex described it as "a human-friendly state machine":

"Even if the system runs completely unattended and automatically, you still need something to track what state it's in and what kind of work is needed. If no humans ever need to look at it, a database is fine — but the moment you want to step in, this is the perfect interface."

Linear's ticket system natively supports state transitions (via labels), event triggers (via webhooks), and historical auditing (via comment threads), making it an ideal intermediary layer between Agent workflows and human decision-making. More importantly, there's long-term value — Linear's data itself is a structured "log of human intent" that future Agents can read to optimize the entire process. The team can analyze how many Issues needed redrafting, categorize them, and build new loops to optimize the original loops. If they find that "a certain type of problem is frequently misclassified as a skill issue," they can add judgment logic to the prompt earlier, so the Agent self-corrects automatically without human intervention.

Loop Two: From Issue to a Mergeable PR

Once an Issue is marked "approved," it automatically triggers a Cursor Agent to create a fix PR. There's a counter-intuitive engineering decision here: don't let Cursor run tests in its own sandbox.

The reason is practical — Cursor's sandbox is "weak and slow," while Boundary's codebase is "quite large." By letting GitHub CI/CD handle tests instead, test time dropped from 15–20 minutes to 2–4 minutes. This reflects a pragmatic systems integration philosophy: CI/CD (continuous integration/continuous delivery) is the foundational infrastructure of modern software engineering, ensuring quality on every code change through automated test and deployment pipelines. Rather than replicating existing capabilities inside an Agent sandbox, having Agents call already-production-ready external services often yields better results.

Once a PR is created, it waits for CodeRabbit's review and CI checks. CodeRabbit is an LLM-based AI code review tool that automatically analyzes code style, potential bugs, and security vulnerabilities after a PR is submitted, providing suggestions as inline comments. If issues are found, another Agent is dispatched specifically to respond to that feedback.

One detail worth noting: the fix workflow isn't a single Agent running end to end — different Agents handle different stages. The first Agent handles the main build; subsequent Agents address CodeRabbit's detailed feedback. These Agents run in containerized environments — interestingly, the team even ran it on a MacBook at one point, triggered by Linear webhooks.

Human Intuition Remains Irreplaceable

A great example is the implementation of array.fill. The Agent's initial implementation had two problems: the parameter was named n (should be length), and it was implemented in BAML rather than using the more efficient Rust standard library function. Dylan simply left a comment saying "use the Rust function and rename n to length," and the Agent made the change automatically.

More subtly, Vaibhav mentioned that their length is an I64 type, but he needed a usize when developing on WASM32. WebAssembly (WASM) is a binary instruction format designed for the web and cross-platform scenarios; its 32-bit variant WASM32 uses a 32-bit memory address space. Rust's usize type is a platform-dependent unsigned integer — on WASM32, it's equivalent to u32 — and I64 (a signed 64-bit integer) would cause a type mismatch compile error or potential runtime truncation on that platform. This kind of detail is something "humans might be too lazy to fix, or might never even think to fix" — which perfectly illustrates the irreplaceable value of human expert judgment in Agent-assisted workflows.

"There are things Agents are really good at, and there are things humans are really good at. Rather than choosing one — either humans write all the code, or Agents write all the code — why not make them symbiotic?"

More Derived Loops: Arena and Automatic Changelogs

The beauty of this system lies in its extensibility. Around the core principle of "have Agents run BAML," the team has spun up multiple services:

• Arena: Give the system a set of candidate skills, test all versions on the same task, and evaluate them by cost, rounds, success rate, and number of issues. The Arena evaluation model comes from the LLM world's "model arena" concept — objectively assessing different candidates by having them compete on the same task. The team found that the best skills were actually the shortest ones — overly long context causes LLM attention to scatter, while BAML describe (BAML's type annotation system) can carry a lot of contextual load, keeping skill documentation concise. This is a prime example of language design and Agent workflows co-optimizing each other.
• Automatic Changelog: Another Agent iterates through each nightly build, finds all commits, and automatically writes the changelog. While the quality of machine-generated logs is inconsistent (formal Canary releases still require human authorship), it's already useful enough as a diagnostic tool.
• Stale Issue Auto-Verification: Another Agent periodically re-runs existing Issues to check whether they've been naturally resolved. Sometimes fixing one problem inadvertently fixes another, and the system can automatically detect this.

The Core Takeaway: You Don't Have to Build the Whole Factory at Once

Perhaps the most important insight from this discussion is a point Dex kept coming back to:

"You don't have to build the entire software factory at once. You don't have to plan out the whole future system. You can start small — even just that tiny loop of redraft, then back to not started. You're constantly stacking and layering these small, isolated problems."

This is a pragmatic methodology that can "start generating value tomorrow," rather than the grand narrative of "I'm going to completely overhaul how we ship software." Find the places where Agents can help, figure out where the human leverage points are and where the Agent leverage points are, then start building and connecting them together.

As Vaibhav put it: "One day you'll wake up and find that 80% of your work has already been automated. It'll probably make the product better faster than you could on your own."

Key Takeaways