Metacache and Reasoning by Construction: The Next Step for Compound AI Systems

From Monolithic Models to Compound Systems

A technical post on Reddit puts forward a perspective worth paying attention to: the AI industry is moving away from monolithic models toward compound systems — architectures where multiple small AI agents each handle a specific task, with an orchestration layer determining how they collaborate. Using the earlier Jev project as a jumping-off point, the article attempts to sketch out the next step in the evolution of reasoning technology.

This isn't an arbitrary claim. The author points out that monolithic models and code are essentially opposites: models "program themselves" during training, but their behavior is unpredictable and hard to modify; code is precise and controllable, easy to edit, but must be explicitly written and doesn't generate itself. The value of compound systems lies precisely in combining the best of both worlds.

Why Monolithic Models Fall Short

The author identifies two practical weaknesses of monolithic models with fairly clear logic.

Unpredictability

The same question might yield a correct answer today and a wrong one tomorrow. This means behavior cannot be fully controlled — a serious flaw for production scenarios that require stable output.

Difficulty of Modification

A model's behavior is baked into its trained weights and cannot be directly edited. While you can "guide" it using prompts, RAG (retrieval augmentation), or fine-tuning, guiding is not the same as setting behavior — and fine-tuning can break parts that previously worked. The only safe fix is to retrain from scratch with new data, which is both time-consuming and expensive.

Code is the exact opposite: it executes strictly as written and can be easily changed by editing it. The trade-off is that it won't generate itself — it must be explicitly written.

RAG (Retrieval-Augmented Generation) is one of the most common ways to intervene in model behavior today: when a user asks a question, the system first retrieves relevant document chunks from an external knowledge base and inserts them into the prompt, effectively letting the model "see" up-to-date or domain-specific information. RAG effectively addresses the model's knowledge cutoff limitation, but it only changes the model's input — it doesn't touch the model's reasoning logic or weights, making it a form of "guidance" rather than "configuration." Fine-tuning, on the other hand, continues training a pre-trained model on a small amount of domain-specific data to improve performance on specific tasks. The cost is the risk of "catastrophic forgetting" — fine-tuning for one behavior may unexpectedly degrade performance on others. Because neural network weights are highly coupled, it's difficult to precisely locate and modify a specific behavior, which is the fundamental reason the author describes modifying a monolithic model as so difficult.

Compound Systems: Getting the Best of Both

Compound systems split work across multiple AI agents, each handling only a narrow task, while a code-based orchestration layer decides which agent runs, in what order, and how results are combined. This yields two key properties simultaneously:

• Customizability: System behavior can be changed by editing the orchestration layer's code, without retraining the model.
• Predictability: Each agent's task is small, making it more predictable than a monolithic model.

The article argues that Jev is one such agent — it returns typed decisions (yes/no, categories, or numbers) rather than free text. For this kind of narrow decision-making, deploying a large monolithic model is wasteful, and a smaller model like Jev is actually more appropriate.

Reasoning by Construction: Letting Models Build Their Own Systems

The article's most central proposal is "Reasoning by Construction." Today's compound systems are all pre-built by humans; the next step is to have the model build one itself during the reasoning phase.

The specific mechanism: when you ask the model a question, instead of answering directly, it constructs a compound system, runs it to compute the answer, and returns both the answer and the system it constructed. The author calls the compound system a model builds for reasoning a reasonlet, and calls a model that works this way an RCM (Reasoning-by-Construction Model). A simple question might produce a reasonlet with only an orchestration layer and no agents.

Using running code to compute answers isn't new — code interpreter tools have been doing this for a while. What's genuinely novel is two things: the system the model builds is preserved rather than discarded, becoming a reusable reasonlet; and this reasonlet is returned to the user along with the answer.

Code Interpreter is a tool capability provided by OpenAI and others in ChatGPT, allowing the model to generate Python code and execute it in real time in a sandbox environment to complete tasks like data analysis, mathematical computation, and chart generation, returning execution results to the user. The core of this capability — "using running code to compute answers" — shares the same starting point as RCM. But the key difference is this: Code Interpreter discards the code after each run, and the model must regenerate code the next time it faces a similar problem; whereas RCM's design goal is to preserve the system (reasonlet) the model constructs, as a reusable reasoning unit — avoiding the computational overhead of repeated construction, and allowing users to directly inspect and modify the reasoning logic itself, rather than only observing the final answer.

Why Reasonlets Matter: The Birth of Metacache

Returning reasonlets creates several key benefits:

• Verifiability: The reasonlet is the exact reasoning process the model used — answers no longer need to be taken on faith.
• Local execution: A reasonlet contains its orchestration layer and the agents it calls, and can be run repeatedly on a local machine with the same or different inputs, without querying the model again. A reasonlet saved and reused this way is called a metacache.
• Modifiability: The orchestration layer is code — you can edit its logic, not just its inputs. A reasonlet reused with modified logic becomes a "higher-order metacache."

Server-Side Caching

The same reuse can happen on the server side. A model provider can save constructed reasonlets, and when a new request matches a previously handled case, run the stored reasonlet with the new input instead of reasoning from scratch. The model does less work and responds faster. Because a single reasonlet can serve any request that fits its workflow, one cached copy can be shared across multiple requests — even from different users. The more general the reasonlet, the more requests it can cover.

The Generality Trade-off and the Readability Problem

The model should judge from the conversation how general the reasonlet needs to be. The author gives an example: if you've been doing all kinds of math operations and then ask "2+2," a more useful reasonlet is one that can compute any mathematical expression, not just one that adds two numbers. If the conversation provides no clues, the user can also directly state their needs.

There is, however, a practical obstacle: the reasonlet's orchestration layer is text-based code. Understanding the reasoning requires reading code; modifying it requires writing code. The value of returning a reasonlet depends entirely on whether users can read and write code easily and efficiently.

The author argues that visual programming can lower this barrier — but only if the visual language is powerful enough. He identifies two limitations: visual languages can't fully replace text code (interacting with systems, networks, and low-level operations still requires text); and most visual languages today are either powerful but limited to a single domain (like games or hardware), or general-purpose but too simplistic. The article mentions a general-purpose visual language called Pipe as a solution — though the author readily admits it's a project he's still developing himself.

Visual programming refers to a programming paradigm that uses graphical elements — such as nodes, connections, flowcharts, or building blocks — to express program logic instead of text-based code. Representative examples include Scratch for children's education, Unreal Engine Blueprints for game development, LabVIEW for hardware description, and the various no-code/low-code tools that have emerged in recent years. The advantage is a lower barrier to reading and writing, allowing users unfamiliar with programming syntax to understand and modify workflows; the limitation is that expressive power is often constrained, making it difficult to handle low-level system calls, complex data structures, or highly abstract logic. The author introduces his own project Pipe here as a "sufficiently general visual language" solution, aiming to address the readability challenge the reasonlet orchestration layer poses for ordinary users — but the project is still in development, and its actual expressive power and usability have yet to be validated at scale.

A Brief Assessment

The article's chain of reasoning is fairly complete: starting from the limitations of monolithic models, it derives the necessity of compound systems, then introduces the concepts of reasonlets and metacache. The core idea of "preserving and returning the reasoning process so it can be reused and edited" aligns with industry trends around interpretability and reasoning caching.

One thing to keep in mind: the article carries a clear promotional undertone — Pipe is the author's own project, and some methods are still in the patent application stage. The concepts introduced (reasonlet, metacache, RCM) are therefore more of a personal framework proposed by the author than established industry standards. Readers may find it useful as a thought framework for thinking about the direction of compound AI systems, but its real-world applicability remains to be demonstrated.