Local-First Agent Platform Architecture: Keeping Vendors Permanently Out of the Loop

Title

Local-First Agent Platform Architecture: Keeping Vendors Permanently Out of the Loop

Meta Description

An in-depth look at TheYgent's local-first agent platform architecture: dual-plane design, logical model IDs, engine manager, agent-as-document, DBOS persistent runtime, and verifiable privacy commitments that keep vendors from ever becoming middlemen.

Article Content

When building an AI agent platform, most visual tools share a common footnote: local models supported — bring your own server. Behind that footnote lies a mountain of work: picking an engine, downloading the correct quantized version, tuning context size to fit machine memory, keeping the server running, then wiring the URL into the builder. More importantly, the builder itself often defaults to phoning usage data home.

TheYgent starts from the opposite direction. The inference engine is part of the product, and the product will never become the middleman between you and your model that you never chose. This article takes a deep dive into its architecture: what the components are, where the boundaries lie, and why they're divided this way.

Dual-Plane Architecture: A Clear Seam

The most important boundary in TheYgent splits the system into two planes.

The inference plane runs models. It carries all the heavy components — engine processes, model weights, model registry, API credentials — and runs wherever you specify: the same laptop, a GPU server on your network, or both.

The control plane orchestrates agents: graph runtime, agent registry and versions, sessions, triggers, MCP servers, retrieval sources, secrets, and run traces.

One rule keeps this separation pure: the control plane never imports the engine. There are no MLX, vLLM, or llama.cpp bindings inside it. It accesses inference only through the inference plane's OpenAI-compatible HTTP API, via a shared client. This has two consequences.

First, the payload boundary is structural, not policy-based. When you talk to a model directly, the browser sends prompts, audio, and images straight to the inference plane. When an agent runs, the control plane — also hosted by you — sends each model call to the inference plane by logical ID. No data passes through a service you don't run, and when data crosses the network to a hosted API, it's because you registered that API.

Second, deployment stays simple. Inference is the only component requiring a separate service: it occupies GPU, holds gigabytes of weights, and is itself an HTTP server. Everything else is a FastAPI application with internal modularity. The same four process types — the UI, the control plane, the inference plane, and an optional persistent worker — can run on bare metal via make up, or on Docker Compose and Kubernetes. Only the wiring differs.

The control plane / inference plane analogy: This separation is similar to Kubernetes' control plane (API Server, scheduler, controller) versus worker nodes (kubelet, container runtime), and to the query planner versus storage engine in database systems. The control plane handles orchestration, state management, and policy decisions; the inference plane handles the actual heavy computation. TheYgent's key constraint — "the control plane never imports the engine" — is equivalent to a compile-time-enforced boundary: if the control plane can't link against MLX, vLLM, or llama.cpp libraries, it's impossible to accidentally call their memory or APIs. This eliminates, at the architectural level, the common loophole of "policy-based boundaries" — verbal promises not to mix things up while the code could still call them.

What OpenAI-compatible HTTP API means: This refers to the interface the inference plane exposes externally following OpenAI's REST endpoints like /v1/chat/completions and /v1/embeddings, along with their JSON request/response format. Any service compatible with this protocol — local vLLM, remote OpenAI, Anthropic's compatibility layer, or your own model server — can be plugged in as an inference backend. The interoperability this provides means the control plane's client code only needs to be implemented once and doesn't change when engines change.

Logical Model IDs: Agents Never Specify Engines

Every model reference in an agent, chat, or benchmark is a logical ID, such as triage-fast. Engine names never appear on the data plane; if a graph tries to call a model named mlx or llamacpp, the run is rejected up front with engine_name_not_allowed.

Binding an ID to the engine actually serving it happens exactly once, on the inference plane's admin API:

curl -X PUT http://localhost:8081/admin/models/triage-fast \
  -H 'Content-Type: application/json' \
  -d '{
    "binding": "llamacpp",
    "source": "local-path",
    "model": "/path/to/model-Q4_K_M.gguf",
    "modality": "chat"
  }'

Binding has only four values: mlx, vllm, llamacpp, or openai-compatible. The first three are engines whose lifecycle TheYgent manages; everything else in the world — hosted APIs, model servers you already have — is openai-compatible plus a URL. Two other dimensions sit alongside it without being merged: source indicates where weights come from (hf, local-path, url), and modality indicates server capability (chat, vision, embeddings, speech-to-text, text-to-speech, image generation).

The benefit of this design: migrating a model from MLX on a Mac to llama.cpp on Linux — or from a local model to a hosted API — requires only changing the registration once. No agent needs editing, and nothing needs republishing. Because engine selection is per-node, a single graph can run local and hosted models simultaneously: a small local model triages all requests, and the router sends only the hard ones to the API.

Engine Manager, Not Model Server

The inference plane doesn't require you to start servers. It starts them itself.

• Lazy startup. Engines spawn on an idle local port the first time their logical ID is invoked, and requests wait for their health check to pass. This is why the first call to a model is always slower.
• Residency cap. Only a small number of engines stay loaded at once — two by default. When a new engine is needed, the least recently used, lowest priority engine is evicted. Engines with requests in flight are never evicted; if nothing can be released, the call fails with no_capacity instead of thrashing repeatedly.
• Idle reclamation. Engines idle for fifteen minutes are torn down; marking a model as keep warm exempts it.
• Abandoned calls are cancelled. If a client disconnects before a non-streaming response arrives, the request to the engine is closed immediately — this matters especially for image generation, where a single render can take minutes.
• One launcher per engine and modality. Chat, vision, embeddings, transcription, speech, and images each map to the correct server program; combinations with no corresponding server are explicitly rejected with a clear error rather than being served incorrectly.

Models are acquired through an in-app Hugging Face catalog that shows, before you download, whether each quantization fits your machine's memory. Installed weights are registered as local files, so loading them never touches the network.

The trade-off between LRU eviction and residency cap: Keeping only a few engines loaded (two by default) is a compromise between memory footprint and cold-start latency. GPU VRAM and system memory are limited resources, and each loaded engine occupies gigabytes of weights. The LRU (least recently used) policy evicts the engine that's gone longest without use, but avoids interrupting in-progress inference via "engines with requests in flight are never evicted" and priority markers. The no_capacity error when nothing can be released is an explicit backpressure signal — refusing new requests rather than slowing all of them down.

The cost of lazy startup: An engine process is only spawned the first time a logical ID is invoked, and the request must wait for the health check to pass. This explains why the first call is always slower — it includes process startup and loading weights from disk into memory/VRAM, which can range from seconds to tens of seconds. To avoid this latency, mark the model as keep warm so it's exempt from idle reclamation.

Agent as Document: A Typed Intermediate Representation

Drag nodes onto the canvas and draw connections. But underneath, an agent is a typed document — an intermediate representation defined once in Pydantic, from which the TypeScript types the editor uses are generated, with a CI check that fails when the two diverge.

The document carries a content hash computed server-side over canonicalized JSON: defaults filled in, keys sorted, layout blocks stripped. Moving a node on the canvas never changes the agent's identity; changing its behavior always does. Publishing produces an immutable, content-addressed version you can call via API, put behind a trigger, or roll back to.

Secrets are never stored in the document. Connections carry references to encrypted storage, so rotating an API key leaves the reference unchanged — and the agent's hash unchanged too.

Persistence: Reliable When It Matters

Agents run on two runtimes. The interactive runtime is an in-process interpreter that streams tokens live; if the process dies mid-run, the run is honestly marked as failed. That's fine when someone is watching. But unattended work — a 3 a.m. webhook, a weekly schedule, an approval waiting for days — needs more: runs that survive crashes and resume from the last completed step without redoing work.

The persistent runtime provides this. It's built on DBOS — an embedded Python library that records workflow state into the Postgres the control plane already has — requiring no new stateful service to deploy. A few rules guarantee correctness:

• One registered workflow. Resumption looks up the workflow by name, so instead of one per agent there's exactly one, which loads the pinned, immutable agent document and walks it.
• Honest guarantees. Completed steps are exactly-once: replay on resume pulls from the log without re-executing, so POSTs aren't re-sent. The step executing at crash time is at-least-once, so HTTP tools support idempotency keys.
• Orchestration does no I/O. Routers, transformers, rule guardrails, and loop control run inside the workflow body and re-derive identical results from logged outcomes — this determinism is what makes resumption work. All I/O happens in logged steps.
• Tokens are side effects. Streams go to subscribers as they arrive; only the assembled answer is recorded, so replayed steps don't re-stream.

Human approvals, subgraphs, loops, and fan-out mappings are persistent-only node types. A human approval node can wait for days and resume after a restart.

How DBOS records workflow state: DBOS is an embedded Python library that persists the result of each workflow step execution (a success return value or exception) to a relational database. When a workflow re-executes after a crash, DBOS queries the log in step order: if a step already has a record, it returns the logged result directly without re-executing that step's code. This achieves "exactly-once for completed steps" — POST requests aren't duplicated due to replay. The cost is that code inside the workflow body must be deterministic: given the same logged inputs, it must derive the same control flow, so routing, loops, and conditionals cannot depend on random numbers, the current time, or external I/O; all I/O must be wrapped as logged steps.

Observability: No Extra Infrastructure Needed

Every run produces spans and per-node inputs and outputs in your own Postgres, which the UI draws as a zoomable waterfall chart: which node ran, for how long, on which model, using how many tokens. What gets captured — and for how long — is configured per agent, with retention and redaction rules. Exporting to an OpenTelemetry collector is a second, optional sink, never required.

A Verifiable Privacy Commitment

"Your data stays private" is easy to say and hard to verify. So instead of a promise, TheYgent publishes the complete list of every connection it makes and when it makes them: Hugging Face when browsing or installing models, the MCP registry when opening the hub browser, and anything you configure yourself — hosted APIs, remote MCP servers, sites you crawl.

The list includes a request that could have been omitted: at startup, the inference plane's model client library fetches a public model price list from GitHub. It carries none of your data and falls back to a bundled copy when offline — but since it's a request, it's on the list. There's no usage analytics, crash reporting, update checks, or license servers, and TheYgent runs fully offline once installed.

The commitment is precise: you own every hop. You can register a hosted API at any time. TheYgent simply refuses to be a middleman you didn't choose.

The practical meaning of a verifiable privacy list: An ordinary "privacy policy" is legal text that users cannot independently verify and that may change with version updates. The network connection list TheYgent publishes is a technical statement: it enumerates every destination in the code that might initiate an outbound connection and the trigger conditions. This lets technically capable users audit it or verify it via firewall rules. In particular, the GitHub request at startup to fetch model prices contains no user data, but is listed because it exists, embodying the principle of "a complete list rather than only sensitive items." The offline fallback to a bundled copy ensures that even if that request is blocked or the network is down, core functionality is unaffected.

Fail Loud, Contract Deliberately

Two habits run through the codebase. Failures are loud: unknown input references, invalid graphs, engine names where logical IDs belong, ports that can't be filled — each fails fast with a named error code, never a silent fallback. Contracts are deliberate: the data plane shape, the agent document envelope, and the four-value binding enum stay frozen until a change is made consciously, named, and tested — even for incremental changes.

How to Get Started

TheYgent offers its source code under the Sustainable Use License, supporting self-hosting on macOS and Linux — bare metal, Docker Compose, or Kubernetes.

• Code: github.com/al2m4n/TheYgent
• Docs: docs.theygent.ai, including complete architecture documentation for contributors
• Watch it run: video demo, from installation to publishing an agent
• Start from available agents: twelve bundled agent templates

One-line Summary

TheYgent builds a local-first AI agent platform with structurally clear data boundaries, self-hosting, and verifiable privacy through dual-plane architecture, logical model IDs, and a persistent runtime.

Paragraph Summary

TheYgent is a local-first AI agent platform whose architecture centers on splitting the system into an inference plane and a control plane: the inference plane runs models and resides wherever the user specifies, while the control plane orchestrates agents and never imports any engine, accessing inference only through an OpenAI-compatible HTTP API. This structural boundary ensures data never flows through services the user doesn't run. The platform uses logical model IDs to decouple agents from specific engines, so migrating or mixing local and hosted models requires only a one-time registration change. Agents are, underneath, typed documents carrying content hashes that distinguish layout changes from behavior changes. The persistent runtime is built on the DBOS library, recording workflow state in existing Postgres to resume from the last completed step after a crash. TheYgent also publishes its full network connection list, has no analytics or telemetry, runs fully offline once installed, and embraces the engineering principles of failing loud and contracting deliberately.