Soei: Building Observability of Evidence and Intent for Multi-Agent Programming

AI programming agents can now independently complete software work over long durations, across multiple steps and sessions. When developers look back at this work, the truly valuable question is not what the model said, but what was actually done, by whom, and whether it was truly completed. A study on Soei (Structured Observability of Evidence and Intent) takes this as its starting point: it first audits the reliability of the existing open-source tool Agenttrail, then uses real session data to quantify the "evidence gap" in agent work, and finally proposes an evidence-centric observability scheme.

Audit of Existing Tools: Three Structural Problems in Agenttrail

The research team verified Agenttrail item by item against source commit e4ba2da, confirming that the nine constants and mechanisms described in its documentation hold at the file and line level, but at the same time found three key defects: the two event pipelines are never unified, causing events from the same session to be scattered across different chains; its Map service has an unauthenticated local mutation surface, meaning local processes can bypass authentication to modify data. This reminds us that before treating agent logs as audit evidence, the credibility of the tool itself must first be verified.

Real Session Data Reveals the Evidence Gap

The study was run on a prototype containing 45 real programming agent sessions: a total of 102,233 records, 20,577 tool executions, and 2,336 interaction segments between humans and agents. With the help of blind-review classifiers (precision 83.3% and 96.7%, respectively), several numbers are worth noting:

• 23.4% (95% CI 17.1–35.7) of code edit segments had no independent check after the final edit;
• 57.8% of to-dos that agents marked complete after editing lacked supporting evidence in the intermediate steps;
• Using a 10-minute time heuristic to judge session handoffs recovered only 6 of the 24 session pairs connected by content lineage;
• 42% of sessions ran longer than 24 hours, and 48.1% of transcripts exceeded the 2 MiB replay limit;
• 1,326 records across 16 sessions contained strings suspected to be credentials.

These numbers point to the same conclusion: an agent claiming "done" and having verifiable evidence of completion are two different things. Long sessions and oversized transcripts also make post-hoc auditing expensive or even infeasible.

In the text, 95% CI (confidence interval) means that under repeated sampling, there is a 95% probability that the true proportion falls within the interval 17.1%–35.7%. The interval is relatively wide because the sample is only 45 sessions, making this a small-sample prototype study; the conclusions should be regarded as directional evidence rather than precise estimates. Similarly, blind-review classifiers refer to automated or human annotators that classify segments without knowing the research hypothesis; their precision (83.3% and 96.7%) determines the credibility ceiling of the above percentages—classifier misjudgments directly propagate into estimation bias in the evidence gap.

The 2 MiB replay limit refers to a hard threshold set by many agent runtimes to control memory and latency on the number of transcript bytes that can be replayed for a single session; beyond that, full replay is no longer feasible, and one must rely on summaries or sampling. This explains why 48.1% of sessions face the risk of information loss during post-hoc auditing, and it is also the direct motivation for Soei's use of a content-free hash ledger rather than full storage.

Soei's Design: Separating "Claiming Completion" from "Proving Completion"

Soei consists of five components:

1. Acknowledgement-aware event schema, giving event records acknowledgement semantics;
2. Hash-chained local ledger, which on the same corpus is only 20% the size of the raw data, stores no content or credentials, verifies at about 126k events/sec, can detect tampering, and supports deterministic replay;
3. A graph structure aligned with W3C PROV, used to express attribution and lineage;
4. Calibrated attribution and lineage inference;
5. Verification layer, strictly separating completion claims from completion evidence.

The core idea of this design is to shift observability from "recording what the model said" to "recording a verifiable chain of facts," while using hash chains and content-free design to reduce storage and compliance risks.

W3C PROV is the provenance data model standard released by the World Wide Web Consortium (W3C) in 2013, used to describe in a standardized way the three core object types—"Entity, Activity, Agent"—and their relationships (such as wasGeneratedBy, used, wasAttributedTo). It provides a common vocabulary for exchanging lineage and attribution information across systems, enabling provenance records produced by different tools to interoperate. Soei aligning its graph structure with PROV means that the attribution and lineage it records can be parsed by external audit tools according to the standard, rather than being locked into a proprietary format.

A hash chain is a data structure that links each record's hash value to the hash value of the previous record: any tampering with historical records causes all subsequent hashes to mismatch and thus be detected. Unlike a blockchain, it does not require distributed consensus and can provide a tamper-evident audit trail while being maintained only locally. Soei's ledger builds on the hash chain with a content-free treatment—storing only hashes and metadata rather than raw content or credentials—so its size can be compressed to 20% of the raw data while also meeting compliance requirements.

Positioning and Open Questions

The study discusses the work alongside concurrent 2026 proposals and, based on its own data, defines an autonomy level with evidence gating and harness metrics. A notable open question is: how to make evidence collection the default behavior, rather than a post-hoc remedy, without excessively increasing agent runtime overhead. For teams building multi-agent programming systems, this line of thinking provides an actionable audit baseline.

One-line Summary

Audits Agenttrail and uses real data to quantify the agent evidence gap, proposing the evidence-centric Soei scheme.

Paragraph Summary

This study first conducts a source-level audit of the open-source tool Agenttrail, finding three key defects including that the two event pipelines are never unified and that the Map service has an unauthenticated local mutation surface, indicating that before agent logs can be used as audit evidence, the tool's own credibility must first be verified. It then quantifies the evidence gap on 45 real programming agent sessions (over 100,000 records and over 20,000 tool executions): 23.4% of code edits had no independent check after the final modification, 57.8% of completion markers lacked intermediate evidence, 42% of sessions exceeded 24 hours, nearly half of transcripts exceeded the 2 MiB replay limit, and there were suspected credential leaks. Based on this, it proposes Soei, using an acknowledgement-aware event schema, a hash-chained local ledger, a W3C PROV-aligned graph structure, and a verification layer to strictly separate "claiming completion" from "proving completion," with the ledger only 20% the size of the raw data and verification at about 126k events/sec.