[KongchangAI]
· 2 min read· 1,044 words

Attest: Zero-Model-Call Replay for Regression Testing LangGraph Agents

Attest: Zero-Model-Call Replay for Regression Testing LangGraph Agents

Attest records LangGraph runs via OTel and replays them in CI, enabling zero-model-call assertions on tool-calling behavior.

Attest is an Apache-2.0 open-source project that tackles the difficulty of asserting on LLM agent regression tests. It records a real LangGraph run via OpenTelemetry spans, then replays it in CI, answering model calls with recorded content so no real model requests are made. Assertions focus on tool-calling behavior, such as call order and per-parameter call-count limits, turning agent behavior into versionable test fixtures. The author shares the LangGraph setup and candidly notes that LangGraph.js capture is not yet available, some adapters are in beta, and most of the code was generated by a coding agent. The approach suits Python teams wanting low-cost CI integration, but is not meant to validate the semantic quality of responses.

Why Agent Testing Always Feels "Unclear"

Writing regression tests for LLM agents has always been awkward. You can't assert on return values the way you would for an ordinary function, because the same input can produce differently worded outputs from the model. You also don't want to actually call the model in CI, because it's expensive, slow, and introduces non-determinism. The common approach in the community is to write a pile of integration tests that "look right," which often ends up degrading into clicking through things manually.

The open-source project Attest (Apache-2.0) tries a different angle: record a real run, replay it in CI without making any model calls, and then assert on tool-calling behavior. Its author recently wired the capture capability into LangGraph and publicly shared the configuration approach and current shortcomings on Reddit.

Recording a LangGraph Run

Attest's capture reads OpenTelemetry spans, and LangSmith itself supports an OTel mode. The key step is pointing the tracer provider at Attest's own local receiver—the LangSmith API key is never used at all, just a placeholder.

The command the author gives is roughly this: set LANGSMITH_TRACING=true, LANGSMITH_TRACING_MODE=otel, LANGSMITH_API_KEY=unused, then run attest capture --name refund-bug -- python run_graph.py.

The recording process keeps two kinds of spans, chat and execute_tool (runnable/chain spans are ignored), and records every model interaction through a local proxy while redacting secrets.

One flaw the author explicitly calls out: the message ids and lc_run-- ids that LangChain generates inside the LangSmith prompt string are not templated, so they show up when you diff two captures. However, the replay matching stage ignores these ids, so it doesn't affect assertion results.

Attest LangGraph record and replay flow diagram

OpenTelemetry (OTel) is an open-source, vendor-neutral observability framework that defines a unified data model and API/SDK specification for three signal types: traces, metrics, and logs. A trace consists of a set of spans, where each span represents an operation (such as a model call or tool execution) and carries a name, attributes, timestamps, and parent-child relationships. LangSmith's OTel mode allows spans generated inside LangChain/LangGraph to be exported via the standard OTLP protocol instead of being sent to the LangSmith cloud. Attest leverages exactly this by pointing the tracer provider at a local receiver to intercept spans, completing the recording without depending on the LangSmith service. Understanding the structure of OTel spans helps clarify why Attest keeps only chat and execute_tool spans—they correspond to model interactions and tool executions respectively, and are the core objects for behavioral assertions.

LangGraph is a framework from the LangChain team for building stateful, multi-step agent applications. It organizes nodes and edges as a graph, supporting loops, conditional branching, and persistent state. Nodes in the graph can be model calls, tool executions, or custom logic, while edges define control flow. LangGraph's Python and TypeScript versions differ in instrumentation support: the Python version can already produce GenAI spans through LangSmith's OTel mode, while the TypeScript version does not yet offer equivalent instrumentation, so Attest's capture path is temporarily unavailable for LangGraph.js. Here, instrumentation refers to the framework's built-in tracing capability, which automatically generates spans that follow semantic conventions for model calls and tool executions—the foundation for record-and-replay.

Replay: No Model Touched in CI

When tests run, you execute the current version of the graph, while the proxy answers model calls using the recorded content—so CI makes zero model calls throughout. Assertions are written at the tool-call level, for example:

chain = (
    expect(trace)
      .tools_called_in_order(["lookup_order", "issue_refund"])
      .tool_cardinality("issue_refund", per="order_id", at_most=1)
)
attest.evaluate(chain)

The meaning of this assertion is: the tools must be called in order, and for the same order_id, issue_refund may appear at most once. This maps exactly to the "duplicate refund" incident the author demonstrates—first record the bug, then run the fixed code in CI to confirm that the duplicate refund is blocked.

If you only want callbacks and don't need full capture, Attest also provides a LangChainAdapter: inside the adapter.capture() context, attach the handler to the callbacks of graph.invoke, then retrieve the trace object from adapter.trace.

Example of tool call order assertion

Record-and-replay testing is a classic testing pattern: first record the inputs and outputs of a real interaction in production or development, then substitute the recorded data for the real dependency during testing. In the agent scenario, model calls are replaced by recorded content, but the current version's graph logic, tool implementations, and routing decisions still execute for real. This validates the impact of code changes on tool-calling behavior while avoiding the cost, latency, and non-determinism of model calls. The replay matching stage usually needs to handle non-deterministic factors (such as random ids and timestamps); Attest's strategy is to ignore differences in these ids and focus only on behavioral characteristics such as tool call order, arguments, and counts. This pattern requires that recording covers enough branches; otherwise, replay can only verify the paths that were recorded.

Clear Boundaries and Shortcomings

The author states the limitations plainly, which deserves credit:

  • The capture path has been tested on Python, with langgraph 1.2.19 and LangSmith 0.14.17.
  • The TypeScript version of LangGraph currently has no instrumentation that produces GenAI spans, so capture for LangGraph.js is unavailable; the LangChain.js callback adapter works but is in beta.
  • The Python LangChain adapter is likewise in beta.
  • The author candidly notes that most of Attest's code was generated by a coding agent based on plans he wrote and reviewed. This is important context for anyone evaluating the project's maturity.

Who This Approach Is For

The value of record-and-replay testing lies in turning "agent behavior" into a versionable asset: a recorded trace is a test fixture, and assertions are an explicit expression of business rules—for example, refunds may happen only once, and a lookup must precede an action. It is not suited to validating the semantic quality of model responses, nor does it solve the question of whether the recording itself covers enough branches. For LangGraph Python users, this is a low-barrier option that can be plugged into CI immediately; TypeScript users will need to wait for instrumentation to catch up.

For teams moving agents from demo to production, this combination of "zero-model-call replay + tool-call assertions" may be one of the most cost-effective forms of regression testing available today.

#Attest#LangGraph#agent regression testing#OpenTelemetry#record and replay#tool call assertions#CI testing
Share:

Related articles