Catalyst: Letting AI Agents Query Compiled Code and Get Verifiable Answers

The Trust Problem for AI Agents

Today's LangChain/LangGraph agents are quite capable when it comes to tool calling — search, databases, code execution, APIs, browsers, you name it. But there's a long-overlooked problem: when a tool returns a number, the agent typically has no choice but to trust it.

The problem gets thornier when an agent needs to modify real engineering software: how does it know what to change? It can read source code, reason about it, and make educated guesses. But that's fundamentally different from actually measuring the program's own behavior. A Catalyst project shared by a Rust developer on Reddit is designed precisely to bridge this gap.

Catalyst isn't another agent framework — it's a deterministic engineering measurement tool for existing agents. When an agent needs to know "what's actually controlling this result right now, how strong is the effect, in which direction, and can this answer be independently verified before I act on it" — it can call Catalyst. This fits naturally with LangChain's existing tool philosophy: callable operations with structured inputs and outputs, invoked by the model on demand.

An Experiment with a Data Center Cooling System

The author demonstrates using an ordinary Rust data center cooling and power model. The system has 52 tunable "knobs": IT load, outdoor temperature, electricity price, fan settings, chilled water settings, recirculation, chiller efficiency, UPS/PDU efficiency, throttling thresholds, carbon costs, thermal risk limits, and more.

The model simulates 72 hours of continuous operation, where each hour's state affects the next — fans react to temperature, room temperature carries forward, filters gradually clog, throttling changes future compute load, chiller efficiency shifts with conditions. The entire simulation produces one number: total operating cost.

Imagine a LangGraph agent tasked with optimizing this system, facing 52 variables it could potentially modify. The naive agent loop typically goes: read source code → judge which variables look important → make changes → rerun. What Catalyst offers instead is: measure the program's actual behavior first.

It does this by reading the LLVM IR already generated by the rustc -O compiler, computing the rate of change (derivatives) of the final 72-hour cost with respect to all 52 inputs across the entire coupled simulation, and independently verifying every derivative before returning results.

In the live demo:

• 52 derivatives
• 105 independent verification points
• 105/105 passed
• Total time: 1.37 seconds (including derivative computation, verification, and reusable artifact generation)

Same Code, Different State, Different Answers

The most interesting results emerge from the "operating point" analysis.

Under heat wave conditions — 38°C, 1900 kW load — the top cost driver is fan_target_c (fan target temperature), followed by IT load, throttle threshold, fan floor, safety temperature limit, airflow, and recirculation. Only 17 of the 52 inputs account for 90% of the measured influence.

When the author ran the same compiled program under mild conditions (18°C, 1200 kW), the important variables changed entirely: UPS efficiency, PDU efficiency, IT base load, electricity price, and peak electricity multipliers took center stage. This time just 9 inputs accounted for 90% of the influence.

The key insight: fan target ranks first under a heat wave, but its measured derivative is 0.0 on a mild day, and throttle control also drops to zero. Same source code, different state, completely different answers.

An LLM reading source code can only see that the fan controller "looks like important code." Catalyst can tell the agent: "Under current conditions, this is the strongest control lever in the system" — or conversely: "Under current conditions, adjusting this control does nothing." For agent systems, this distinction is enormously valuable.

What the LangGraph Loop Looks Like

The author's envisioned graph structure is: inspect code → call Catalyst → rank controllable inputs → agent proposes changes → human approval if needed → apply changes → rerun verified artifact → run regression drill → report evidence.

LangGraph is already designed for workflows that mix deterministic steps with agent decisions, persistence, and human-in-the-loop control. Catalyst serves as the deterministic measurement and verification step within that structure, rather than trying to replace the orchestration layer.

The demo actually executed a simplified version of this decision loop: take the strongest controllable inputs, stay within their declared bounds, move in the cost-reducing direction indicated by the measured derivatives. Modeled cost went from 40,033.55 → 34,355.69 → 32,769.23, a reduction of 18.1%.

But the author emphasizes that more important than the reduction is why those variables were chosen — they were selected because the compiled program measured them as the dominant control factors, not because an LLM thought their names sounded important.

Once the first verified artifact is generated, it can be quickly evaluated at another point. Benchmarks show artifact re-evaluation takes about 12 milliseconds. So the agent can do: careful verified measurement → exploration → move → re-evaluate → decide whether a full new measurement is needed — without blindly recomputing everything from scratch.

Structured Tool Interface and Verification Boundaries

Catalyst's tool interface is structured rather than prompt-based, exposing validated operations like discover, validate_problem, evaluate, export_go, and differentiate_llvm. A discover call describes other tools, their schemas, rejection codes, and remediation options, so tool-capable agents can self-discover Catalyst's capabilities and receive machine-readable success or failure rather than having to parse terminal output.

This means a Catalyst call can return verified, or return refused because the derivative disagrees, rather than forcing the LLM to judge whether a suspicious number "looks right."

The author also deliberately built in one hard boundary: agents cannot modify Catalyst's verification tolerances or redefine what counts as success. The model can request measurements, but it cannot approve its own measurement results. This allows Catalyst to sit behind an agent without letting the agent act as both player and referee.

Artifacts Can Flow Between Multiple Agents

Derivative results include provenance information covering source code, compiler, LLVM IR, raw computation, derivatives, and verification. In the demo, the author modified a single byte in a stored derivative — Catalyst refused to run and returned catalyst.artifact_tampered.

This enables an interesting multi-agent pattern: Agent A measures something, Agent B later receives the artifact, and B doesn't have to accept "A says this was verified" at face value — the artifact can be re-examined.

Results don't have to stay in Python or Rust either. The demo also exported a verified chiller calculation as standalone Go and base R implementations, both derived from the same optimized computation and validated against 26 fixture cases. A LangChain agent could theoretically orchestrate: existing Rust/C/C++ code → Catalyst analysis → verified Go implementation for services → verified R implementation for analysts, with LangGraph still handling orchestration, state, and approvals.

Part Two: Proving a Fix Actually Works

The demo also includes a local ticket service that runs normally under ordinary conditions but starts returning POST /orders -> 503 under sustained burst load.

Catalyst reproduced the failure from a 96-request run, reduced it to a 16-request reproducer, generated held-out scenarios, and compared baseline against a candidate fix:

• Baseline: 9 errors out of 16, p99 ~245 ms
• Candidate: 0 errors out of 16, p99 ~16 ms, 3/3 held-out scenarios passed

This is especially relevant for coding agents. A LangGraph agent can propose a fix, and Catalyst gives the graph a deterministic gate: does the original failure still reproduce? — rather than letting "tests look green" be the end of the workflow.

Positioning Relative to LangSmith

There's a lot of community discussion around debugging multi-step agents, tool failures, and "the final answer looks fine but an early tool call was actually wrong." Catalyst targets a slightly different layer of that problem:

• LangSmith can tell you what the agent did;
• Catalyst helps you confirm whether the numerical results or engineering changes the agent acted on actually have independently verified evidence behind them.

The simplest positioning statement: LangChain/LangGraph decides when to act; Catalyst gives agents engineering measurements they don't have to invent or blindly trust.

Potential use cases the author lists include: coding agents modifying numerical Rust/C/C++ systems, scientific/data agents needing verified derivatives rather than LLM-estimated relationships, infrastructure agents determining which configuration variables actually drive metrics, passing verified computations between agents in multi-agent systems, and LangGraph workflows requiring deterministic approval gates.

The project is open source (repository link in the original post). The author also poses an open question to the community: for a deterministic tool like this, which form is most useful — as a standard LangChain tool, as a LangGraph verification node, or as a gate that must be cleared before an agent is allowed to apply changes?