Agent Trace Sampling: Preserve Failure Traces at Run Granularity

The Sampling-Level Mistake: What You Throw Away Is Exactly What You Need Most

At 09:12 in the morning, the on-call page went off. A settlement Agent had been returning HTTP 500 for forty minutes straight.

Opening the trace store, there were two weeks of Span data inside. The run that failed was not among them.

The collector kept the boring Spans and discarded the valuable ones. This is a classic sampling anti-pattern, and at its core it's a data retention strategy problem, not a tracing capability problem.

Most teams sample at the Span level, using a coin flip to independently decide whether each Span is kept. This directly breaks causality — the tool call is retained, but its parent LLM call isn't, and you can't replay a trace with holes in it.

Worse, failures are rare and correlated. A 5% Span sampling rate captures roughly only 5% of the content of a failing run, and the leaf Spans that carry the real diagnostic value happen to be exactly the ones already flagged by status filters. You pay 5% of the storage cost but lose 95% of your debuggability. The incident you care most about is precisely the one you store the least of.

Span vs. Run: A Conceptual Distinction

In distributed tracing, a Trace represents one complete end-to-end request flow, a Run typically refers to one complete Agent task lifecycle (e.g., a user session or a tool-orchestration cycle), and a Span is the minimal unit of work inside a Trace, corresponding to one specific operation call (e.g., one LLM inference, one tool call, one database query). A single Run often contains dozens or even hundreds of Spans, connected via parent_id into a tree-shaped causal structure.

The key engineering implication is this: the sampling granularity defines the boundary of replayability. If you sample independently at the Span level, when a parent Span is dropped but its child Span is kept, the causal chain has a hole — you see a tool call but not the LLM decision that triggered it. A Run, by contrast, is a natural atomic unit: it has a clear start and end, a clean status, and a stable identifier (run_id). Treating the Run as the sampling unit means the retention decision is a single semantic judgment, not a coin flip per operation. This is exactly the premise behind the "treat the Run as a whole" design below.

Changing the Sampling Unit: Treat the Run as a Whole

The solution is to raise the sampling unit from Span to Run. A run is either fully recorded or fully summarized — there is no partial run, and there are no orphan Spans.

The author proposes three storage modes, which are enough to cover the scenarios:

• Full: every Span, every payload, exact bytes, and sequence numbers are all preserved.
• Summary: one lossy record per Span, containing ID, type, status, duration, token count, byte count — no payload.
• Anomaly-pinned: identical to Full, but permanently exempt from sampling, because the run has already proven it can go wrong.

On top of this, add one rule that overrides randomness: any run touching a known anomaly retains full detail regardless of the sampling rate.

The Engineering Positioning of the Three Storage Modes

These three modes are not mutually exclusive options, but rather layered outputs of the same policy under different run states.

Full mode plays the role of "forensic evidence" — it must preserve exact bytes and sequence numbers, because incident review requires byte-by-byte payload comparison, execution order reconstruction, and timestamp causality verification. Any lossy compression makes replay meaningless.

Summary mode plays the role of "statistics and trends." It retains scalar fields such as ID, type, status, duration, token count, and byte count — enough to answer "is latency rising over this period," "which tool's call volume spiked," "how has the cost distribution shifted," but not "what exactly were the parameters of that call." It's cheap, so it can cover the vast majority of clean runs and be retained long-term.

Anomaly-pinned mode is the safety net of the whole strategy. It doesn't depend on the sampling rate; instead, it's triggered by the run's own historical evidence (an error or retry status appeared) to grant a permanent exemption. Here "permanent" is relative to the sampling rule — the run is still subject to the retention period (e.g., 30 days in the decision table), but won't be randomly discarded by the sampling rate. It's important to understand this: pinning addresses "whether it's selected," while the retention period addresses "how long it's kept" — the two are orthogonal dimensions.

A Decision Table You Can Argue About

Span or Run Category	Decision	Retention	Reason
Run contains any error Span	full	30 days	Root-cause evidence
Any Span with retry status	full	30 days	Retry storms hide real bugs
Span with truncation flag	full	14 days	Payload already truncated
Span exceeds latency p99 budget	full	14 days	Regression evidence
Clean Run, hash-selected	full	7 days	Baseline for Diff
Clean Run, not selected	summary	90 days	Trend lines and cost accounting
Health-check Span	not stored	0	Pure noise

There are two counterintuitive aspects to this table. Clean runs are kept longest in summary form, because they're cheap and can answer trend questions; failing runs are kept shortest in full form, because the fix will come quickly.

Sampler Implementation: About Forty Lines of Code

The following is a proposal, not a result benchmarked in your environment. Before using it on production traffic, run it against archived trace data first.

**The Role of Hash Sampling and SALT**

The core of the `selected_for_full` function is `hashlib.sha256((SALT + run_id).encode())` followed by modulo-based bucketing. This deterministic hash sampling has two irreplaceable properties.

First is **idempotency**: the same run_id gets the same decision in any process, on any retry, at any point in time. If you used `random.random()` for independent random sampling, then the same run could be judged as kept on the first pass and discarded on the second pass after a collector process crash and restart, ultimately leaving the run partially stored — which is exactly the root cause of the "orphan Spans" this policy aims to eliminate.

Second is **unpredictability**, and this is precisely the point of SALT. Without SALT, anyone can use a public hash algorithm to compute whether a given known run_id is retained. On shared storage, multi-tenant systems, or external debugging platforms, this lets attackers or careless engineers specifically determine "is this run worth keeping," or even reverse-engineer which runs entered full mode. SALT turns a publicly verifiable algorithm into a decision function only the operators know. The uniformity of the hash guarantees it doesn't introduce sampling bias. Note that SALT must be injected via an environment variable, not hardcoded; otherwise it too will be leaked through the source code, and both security and reproducibility will fail simultaneously.
# sampler.py — proposal code; measure on your own archive first
import hashlib
import os

SALT = os.environ.get("TRACE_SAMPLE_SALT", "change-me")
ANOMALY_STATUSES = {"error", "retry", "truncated", "budget_exceeded"}

def is_anomalous_span(span: dict) -> bool:
    if span.get("status") in ANOMALY_STATUSES:
        return True
    return bool(span.get("truncated"))

def run_is_pinned(run: dict) -> bool:
    return run.get("anomaly_count", 0) > 0

def selected_for_full(run_id: str, success_sample_rate: float = 0.05) -> bool:
    """Deterministic selection based on run_id. Samples the entire Run, not individual Spans."""
    digest = hashlib.sha256((SALT + run_id).encode()).hexdigest()
    bucket = int(digest[:8], 16) % 10_000
    return (bucket / 10_000.0) < success_sample_rate

def store_mode(run: dict) -> str:
    if run_is_pinned(run):
        return "full"
    if selected_for_full(run["run_id"]):
        return "full"
    return "summary"

Two design choices are worth noting.

First, hash on run_id, not random lottery. The same run gets the same decision in every process, on every retry, after every restart. Random lottery would make the retention policy non-reproducible.

Second, read SALT from an environment variable. Without it, anyone can compute which runs you kept; with it, the selection is both stable and unguessable.

The Only Test That Matters

One invariant protects the entire policy: anomalous runs are never summarized away.

**Why the Third Test Is the Most Subtle**

`test_selection_is_not_all_or_nothing` asserts that roughly 5% of ten thousand run_ids (between 300 and 700) are selected. Its value isn't in verifying the exact sampling rate, but in detecting **silent policy failures**.

A common bug pattern is this: a SALT change causes all hashes to land in the same bucket range, or the modulo base gets mistakenly changed from 10000 to 1, or the float division in `bucket / 10_000.0` gets replaced with integer division — none of these changes throw an exception, none of them fail any other test, yet they silently push the sampling rate to 0% or 100%. 0% means you lose all baselines and the Diff tool has no data to work with; 100% means storage costs instantly multiply twentyfold. Either extreme only gets discovered at the next incident review, or when the bill arrives.

`test_selection_is_stable_across_calls` is the guard for deterministic hashing: it calls the same run_id 100 times and asserts the result set has size 1. If someone later replaces the hash with `random`, this test fails immediately. Together the three tests cover the policy's three key properties — pinning, reproducibility, and correct distribution — and they add up to under twenty lines of code, which is exactly why they belong in CI.
# test_retention.py — proposal; wire into CI
def test_anomalous_runs_are_never_summarized():
    for status in ("error", "retry", "truncated", "budget_exceeded"):
        run = {"run_id": "r-1", "anomaly_count": 1, "status": status}
        assert store_mode(run) == "full"

def test_selection_is_stable_across_calls():
    a = [selected_for_full("r-42") for _ in range(100)]
    assert len(set(a)) == 1

def test_selection_is_not_all_or_nothing():
    kept = sum(selected_for_full(f"r-{i}") for i in range(10_000))
    assert 300 < kept < 700  # roughly 5%, reasonable variance

The third test is the most subtle. A Salt or modulus bug can silently push the sampling rate to 0% or 100%, and no one will notice until an incident review.

Where the Pipeline Runs

The collector and summarizer are both small, periodic, and CPU-intensive. This fits free server offerings well, since you don't pay for idle capacity between tasks.

The author discloses in the article that it is part of a MonkeyCode product promotion. MonkeyCode is an open-source project offering free model access, with the operator listing 10 million free tokens and a free server option. Verify current terms before use.

Model access has a concrete role here. Storing full payloads is expensive; a one-line run summary is not. So generate a summary for every run, not just the sampled ones, and store it permanently alongside the summary records. Roughly like this:

Input: 40 summary records for run r-8891
Output: one sentence — what the run attempted, what changed, what failed

This summary is the first thing the on-call engineer reads at 09:12. It costs a few hundred tokens per run, which is exactly why free credits can change what you can afford.

Two caveats: don't put payload content you can't share into third-party model calls; and don't treat the free tier as a system of record. Check the current limits and retention terms yourself.

Measure Before You Trust the Policy

Don't adopt this scheme on a hunch. Compute the ratios over one day of archived traces:

# Total bytes and run count
wc -c < traces.jsonl
wc -l < traces.jsonl

# Bytes occupied by anomalous runs (approximation: runs with error lines)
jq -c 'select(.status == "error") | .run_id' traces.jsonl \
  | sort -u > pinned.txt
rg -F -f pinned.txt traces.jsonl | wc -c

Then report two numbers: pinned_runs / total_runs and kept_bytes / total_bytes.

The author's suggested acceptance criteria are below; you can substitute your own:

• pinned_runs / total_runs below 2% in steady state.
• kept_bytes / total_bytes below 15% after summarization.
• Zero runs missing a stored summary.

If pinned_runs is far above the error rate, your anomaly rules are too broad. Fix the rules first, then lower the storage bill.

Limitations and Boundaries of Applicability

• Summaries can't answer unanticipated questions. Fields not recorded won't be found later. Only add fields to the summary schema when you can justify the byte cost.
• Sampling breaks exact replay of unselected runs. The sampling rate must be high enough to ensure Diff baselines exist.
• Hash sampling can correlate with ID schemes. If run_id starts with a tenant prefix, some tenants get sampled more. Salt alone can't fix this — you need to hash the full ID or re-bucket.
• The latency budget needs a warm baseline. On a sleeping free server, the first request after idle inflates the duration. Exclude cold-start Spans, or the p99 rule will pin everything.
• Not for audit or legal hold. A sampling policy is an engineering choice, not a compliance record.

If your Agent runs fewer than a few thousand times a day, storage is cheap, full tracing is simpler, and you can skip the sampler. If you're bound by retention obligations, keep everything and solve the cost elsewhere. If your failures happen to be the kind that would be sampled away — say, silently wrong answers with a clean status — pin based on outcome signals rather than status codes.

Start With One Day of Traffic

Sample runs, not Spans. Pin anomalies unconditionally. Store a model-generated summary for every run.

Take one day of archived traces, run the sampler against it, and report the two ratios and the three invariants. If the numbers hold, you've shrunk storage without losing the very run you needed at 09:12.

One-line Summary

Raise the trace sampling unit from Span to Run, unconditionally pin anomalous runs, and avoid losing the failure scene you need most.

Paragraph Summary

The article points out that independent sampling at the Span level breaks trace causality and produces orphan Spans that make the trace unreplayable; failures themselves are rare and correlated, so a 5% sampling rate typically only captures 5% of the content of a failing run — precisely the highest-diagnostic-value Spans, which are flagged by status filters but still randomly discarded. The solution is to raise the sampling unit from Span to Run, using three storage modes — Full/Summary/Anomaly-pinned — combined with deterministic selection based on run_id hashing and an unguessable SALT, ensuring each run is either fully recorded or fully summarized. The core invariant is "anomalous runs are never summarized," enforced by CI unit tests, which must also detect silent bugs where the sampling rate degenerates to 0% or 100%. The author suggests first measuring two ratios against one day of archived data — pinned_runs/total_runs (<2%) and kept_bytes/total_bytes (<15%) — before adoption.