Seahaven: An Open-Source Framework for Building RL Training Environments for LLM Agents

When training LLM agents with reinforcement learning, the hardest part is often not the algorithm — it's the environment. A developer with over a decade of AI optimization experience at Apple, their own startup, and Kiln has open-sourced a Python framework called Seahaven, designed specifically for building reproducible RL training environments for LLM agents. The project name comes from the fictional town of Seahaven in The Truman Show, symbolizing the idea of crafting a controlled, artificial world for an agent to inhabit.

Why the Environment Is the Hardest Part

For multi-turn agents that need to call tools, building a training environment is far more complex than it might seem. The author points out that every episode must satisfy several demanding requirements: realistic data, stateful tools (a write in turn 3 affects a read in turn 30), and the ability to reset precisely to the same starting point — all while running thousands of instances in parallel.

The author previously built several large RL environments by hand and concluded that "it's doable, but hard." More critically, every new environment meant rebuilding the same underlying infrastructure: per-episode isolated databases, frozen starting states, parallel instances, clock control, and logs capturing every state change. This repeatedly needed "foundation" is exactly what motivated the creation of a general-purpose framework like Seahaven.

In reinforcement learning, an episode refers to the complete sequence of an agent starting from an initial state, interacting with the environment, and reaching a terminal condition. For tool-calling LLM agents, a single episode may involve dozens of rounds of "agent issues a tool call → environment executes and returns a result" interactions. Unlike game-based RL environments, these agents often involve persistent state (database records, filesystem changes, etc.), making reproducibility extremely difficult: if a database write in step 5 affects a query in step 20, every reset must restore the exact same initial database state — otherwise training samples become incomparable, and reward signals become noisy due to differing initial conditions. This is the fundamental reason Seahaven treats "millisecond-level reset to a fixed starting point" as a core capability.

What Seahaven Does

Seahaven is an MIT-licensed open-source framework whose core purpose is to "build synthetic worlds for LLM agents" — working replicas of the tools and data an agent operates on. Developers only need to write the logic specific to their world; the framework handles the rest. Crucially, every Seahaven world is an OpenEnv environment, compatible with TRL's OpenEnv support or any OpenEnv-compatible trainer.

The framework handles the following core capabilities:

• Stateful: Each instance has its own isolated SQLite database. Agent tool calls read from and write to it — actions are tool calls, observations are call results.
• Fixtures: You can freeze starting states like small_startup or big_co and reuse them across episodes. Reset delivers a fresh copy in milliseconds.
• Parallel: Each connection gets an independent instance; a single process can host hundreds of them.
• State-based rewards: Every row the agent modifies is logged with before/after values, letting you "grade outcomes, not trajectories" — enabling verifiable rewards without needing an LLM judge for most tasks.
• Deterministic: The same fixture, clock, and random seed produce identical episodes. Clocks and seeds are wired into the SQLite connection, so even random() and CURRENT_TIMESTAMP in SQL queries replay exactly.
• Composable: Your world can embed other worlds, reusing their tools and APIs.

OpenEnv is a standardized RL environment interface specification — analogous to OpenAI Gym (now Gymnasium) in traditional RL — designed specifically for language model agent scenarios. It defines standard methods like reset(), step(), and state(), allowing different training frameworks (such as TRL, veRL, etc.) to interact with an environment without knowing its internal implementation. TRL (Transformer Reinforcement Learning) is Hugging Face's open-source library for fine-tuning language models with reinforcement learning, supporting algorithms like PPO and GRPO. By targeting the OpenEnv interface, Seahaven means developers don't need to rewrite environment adapters for different trainers — a single Seahaven world can plug directly into any compatible training pipeline, dramatically reducing engineering migration costs.

Stripe World: A Strikingly Realistic Example

To showcase the framework's capabilities, the author built a "fake Stripe" example world called Stripe World. It contains 24 tables and 155 API operations, exposing tools identical to Stripe's official MCP server. Even more impressively, it implements Stripe's REST API with such fidelity that the official Stripe SDK runs on top of it without any modifications.

This example demonstrates Seahaven's practical value: you can build high-fidelity training sandboxes for complex real-world SaaS tools, letting agents safely and reproducibly learn in them without worrying about polluting real data or triggering real charges.

What an Episode Loop Looks Like

At the code level, building a training loop with Seahaven is remarkably clean:

from seahaven.openenv import SeahavenClient

with SeahavenClient(base_url="http://127.0.0.1:8000") as env:
    for episode in range(100):
        env.reset(fixture="big_co", seed=episode)  # Fresh private world in milliseconds
        run_policy(env)  # Agent calls tools: env.call("create_lead", ...)
        reward = grade(env.state())  # Every row the agent touched is traceable

The flow clearly reflects the RL training paradigm of "reset to a fixed starting point → run policy → score based on final state."

Toolchain and Ecosystem Compatibility

Seahaven also demonstrates solid engineering completeness. It provides both synchronous and asynchronous clients, and worlds can be published to Hugging Face. The seahaven mcp command serves a world to any MCP client, letting you manually test-run tasks before committing to a policy; seahaven serve provides a web console where you can open instances, call tools, and inspect state in the browser.

The entire system runs fully locally, requiring only Python 3.14+ and SQLite — no external services needed. The framework also includes built-in documentation, a linter, and tests designed for AI agents to build worlds with, aiming for a development experience "optimized for agents."

The author notes that Seahaven was born at Kiln (their team) and is used internally for evaluating and optimizing agents, but it is a standalone project that requires no Kiln dependency. They also raise several directions for future development — reward helper utilities, fixtures as curriculum learning, and more example worlds — and are openly soliciting community feedback on training scenario adaptations.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic in 2024, designed to standardize how LLMs connect to external tools and data sources. Through MCP, any compliant client (such as Claude Desktop, Cursor, or other IDE plugins) can discover and invoke tools exposed by a server without writing custom integration code for each tool. Seahaven's built-in seahaven mcp command publishes a training environment directly as an MCP server — a design that lets developers manually "play" with the synthetic world using an interactive MCP-compatible client before launching RL training, verifying that tool behavior matches expectations and significantly shortening the debugging cycle. Curriculum Learning refers to arranging training samples from easy to hard; Seahaven's Fixtures mechanism is naturally suited to this paradigm — starting states of varying difficulty can serve as different curriculum stages, progressively increasing the task complexity an agent faces.

Summary

Seahaven targets a long-underestimated pain point in LLM agent RL training: environment infrastructure. Through OpenEnv compatibility, deterministic replay, millisecond-level resets, and state-based verifiable rewards, it abstracts the foundational work that every team previously had to rebuild from scratch into a reusable framework. For researchers and engineers exploring reinforcement learning with tool-calling agents, this is an open-source project well worth watching.