Reef Infra: Driving AI Optimizer Harness Changes with Explicit Requests

Why Trace Data Alone Isn't Enough to Drive All Changes

In automated AI optimization systems, a common assumption is that the system detects problems by observing failure traces from usage, then makes adjustments accordingly. This pattern works well for recurring failures — traces clearly expose the pattern of the same error appearing repeatedly.

But there's a structural blind spot here. Trace data can only reflect behavior that has already occurred. It cannot capture a new requirement that someone just decided to adopt. When a team agrees on a new output schema, waiting for users to accumulate enough failures under the old schema before the system "notices" the need to change is an absurd way to communicate intent. A requirement is a deliberate decision, not the natural byproduct of repeated failures.

This is precisely the problem Reef Infra aims to solve: it provides two modes for custom harness methods — manual and hybrid — making "proactive requests" a first-class input for triggering changes.

Usage traces refer to the input/output logs, error messages, and execution paths automatically recorded by a system at runtime. They serve as the core feedback source for mainstream LLM optimization frameworks like DSPy and TextGrad. This type of data is naturally suited for discovering "defects in existing behavior" — for example, unstable answer formatting for certain query types, or a reasoning step that frequently fails — because these problems accumulate into statistically significant signals over time. However, the effectiveness of trace data depends entirely on the premise that "the problem has already occurred." In fast-moving product environments, teams often need to complete system adjustments on the same day a new requirement is introduced, rather than waiting for old behavior to accumulate enough failure samples. This temporal mismatch — requirements come first, data comes later — is the fundamental limitation of purely trace-driven architectures, not an implementation detail that can be patched at the engineering level.

The Division of Labor Between Manual and Hybrid Modes

The core of Reef Infra's design lies in distinguishing two fundamentally different types of input sources.

Manual Mode: Pure Intent Input

In manual mode, the proposer receives a queued request along with an empty sample list. In other words, the change is entirely intent-driven, with no dependency on any recent execution context. When you have a clear, newly agreed-upon requirement, a manual request can directly initiate that change — no failure cases required as a prerequisite.

Hybrid Mode: Combining Intent and Observation

Hybrid mode goes a step further. It attaches a subset of samples from the next automatic batch, so the request arrives carrying recent execution context. This means the proposer can see both "what change someone wants" and "how the system has actually been running recently." This design gives the custom method two inputs with distinct responsibilities: observed behavior and an intentful request to change it.

The value of both modes is that they decouple "new requirements" from the report stream. You no longer need every new requirement to masquerade as a series of failure cases before the system will accept it.

The Explicit Contract: The requests Keyword Argument

At the engineering level, Reef Infra establishes a clear explicit contract: custom methods must declare a requests keyword argument.

There's a subtle but elegantly designed detail here that's easy to overlook — a generic **kwargs is not sufficient. If you attempt to use **kwargs to broadly accept parameters, the recipe will reject the configuration at startup.

# Not accepted:
def my_method(**kwargs):
    ...

# Accepted:
def my_method(requests=None, samples=None):
    ...

Fail-fast is a defensive design principle in software engineering that advocates for systems to immediately terminate and report errors upon detecting configuration or interface violations, rather than continuing to run with latent problems. The opposite approach — "silent failure" — allows errors to lurk deep within a system, eventually surfacing as hard-to-reproduce runtime exceptions with high debugging costs. In composable optimization pipelines, interface contracts between components are especially critical — an upstream component quietly dropping a field might not trigger a visible error until dozens of steps downstream. Requiring explicit declaration of the requests parameter rather than accepting a broad **kwargs moves interface compliance checking to the startup phase, strictly separating "configuration errors" from "runtime errors." This is especially valuable in team settings where multiple people integrate custom optimizers, effectively preventing hidden errors caused by inconsistent parameter naming conventions.

From a developer experience perspective, this is "a useful failure to get early." When integrating your own optimizer, the system explicitly tells you at startup that the contract isn't met, rather than letting the problem lurk until it explodes in a hard-to-debug way at runtime. This fail-fast design significantly reduces debugging costs when building composable optimization pipelines.

Evaluation and Publication Strategies Still Hold the Wheel

One important clarification: a manual request can initiate a change, but that doesn't mean it bypasses the system's governance mechanisms. The existing evaluation and publication strategies still determine what ultimately happens to these proposed edits.

This is an important boundary in the power structure. A request simply opens the door to change and gives the custom method a proactive starting point — but whether a change actually takes effect still passes through the established evaluation process. This ensures that requirements can enter the system promptly and directly, while subjective intent never overrides quality control.

In automated AI optimization systems, evaluation and publication strategies typically form the final gatekeeping layer for changes to take effect. Evaluation strategies handle quality measurement of proposed edits — common approaches include running benchmarks on a held-out set, A/B comparisons with the current version, or preference judgments through human annotation. Publication strategies determine under what conditions an evaluated change gets written into the production configuration — for example, "must not regress on any core metric" or "requires human approval." Separating the request entry point from the publication gate is a classic separation-of-concerns design: the request layer is only responsible for expressing intent, the governance layer is only responsible for quality control, and neither oversteps into the other's domain. This architecture lets teams independently adjust their respective strategies — for instance, relaxing the publication gate during experimentation to accelerate iteration, without modifying the request intake logic.

Implications for AI Engineering Practice

The philosophy behind Reef Infra's mechanism has broader reference value for AI systems engineering.

Many current automated optimization and self-improvement systems over-rely on feedback data, defaulting to the assumption that "all changes should come from observation." But in practice, product decisions, protocol upgrades, and schema changes are often proactive choices made ahead of the data. A mature system architecture should provide clear, distinct channels for both "data-driven corrections" and "intent-driven changes," rather than forcing all requirements through the same report stream.

By making requests an explicit first-class input, complemented by a strict interface contract and an inescapable evaluation strategy, Reef Infra demonstrates a more pragmatic balance: respecting human proactive intent while not abandoning the system's quality assurance. For engineering teams building custom optimizers or harnesses, this is a design pattern worth learning from.