Durability Testing for Local AI Agents: The Runtime IS the Real Benchmark

Why Test the "Runtime" Instead of the Model?

Most AI benchmarks focus on the model itself — accuracy, inference speed, token throughput. But when you deploy a model as a local agent that needs to run continuously for days at a time, what actually determines success or failure is rarely the model. It's the entire runtime environment surrounding it: whether retrieval holds up over time, how memory scales, whether the sandbox stays under control, and whether context behavior remains stable.

A developer published a durability benchmark for local agents on Kaggle, making a sharp central argument: the runtime itself is the benchmark. Using a Gemtek NUC M6 Ultra mini PC running Pop!_OS with zRAM swap enabled, they specifically tested sessions that needed to run continuously for days. This wasn't about measuring how fast the model runs — it was about how long the entire system holds together.

Three Model Tiers, One Shared Runtime

The tester chose models they actually run locally, fixing the same runtime and test procedures across three price tiers: a local 1.2B triage model, a local 8B mid-tier model, and a cloud-hosted primary model.

Critically, all data came from real agent sessions, and every number was cross-validated at least three ways before being published. This "same environment, different tiers" setup also became the source for the fifth finding — different tasks simply don't require the same tier of compute.

Five Core Findings on "Durability"

1. Memory Systems Have a Stall Threshold — the Bottleneck Is File Count, Not Byte Size

This is the most counterintuitive finding. The tester discovered that memory stores stall context assembly when approaching roughly 1,500 files per store — the trigger is file count, not data volume.

The solution was sharding: splitting data into 80 bounded stores, achieving linear performance at 2.6ms/file, keeping the largest store 38% below the stall threshold, and delivering a 3.1× speedup overall compared to a single monolithic store. For any developer building long-term memory systems, this is an architectural constraint that must be planned for upfront.

The underlying cause relates to how file systems and vector retrieval indexes are implemented. Many local agent memory systems (such as those based on SQLite, LanceDB, or local vector databases) see their index scan or directory traversal complexity degrade from near O(1) to O(n) or worse as file count grows. Individual file size affects I/O bandwidth; file count affects metadata operations (inode lookups, directory entry traversal, file descriptor allocation). On Linux, ext4/xfs file systems see readdir() syscalls slow down noticeably with thousands of small files in a single directory. During context assembly, agents typically need to open and read many small files concurrently — exactly where this bottleneck hits. The sharding solution distributes files across multiple subdirectories, keeping each partition's file count below the threshold and maintaining metadata operations in the efficient range.

2. Turning Sandbox "Safety" From a Boolean Into a Score

Traditional sandbox testing asks one question: "Did it stop the fork bomb?" — a yes/no answer. The tester argued this is far too coarse to describe real containment capability.

Instead, they switched to a blast radius score: MB of memory pressure, PID-seconds survived, and delta host RSS under zRAM. Safety isn't a binary switch of "blocked or not blocked" — it's a quantifiable degree. This framing makes sandbox containment evaluation much more reflective of real production risk.

3. U-Shaped Recall Curve: Information in the Middle Gets Ignored

After repeated testing on their own documents, the tester validated a widely discussed phenomenon: facts positioned in the middle of the context are used by the model at significantly lower rates than those at the beginning or end — the classic U-shaped recall curve.

Based on this, they changed their writing habits: all internal documents now use a "conclusion first, evidence last" structure. Because they work with information-dense documents, this ensures the agent has sufficient contextual grounding before diving into deeper retrieval. This is a textbook example of translating a benchmark finding directly into a workflow improvement.

The U-shaped recall curve (also known as the "Lost in the Middle" effect) was systematically documented by Stanford researchers in the 2023 paper Lost in the Middle: How Language Models Use Long Contexts. The study found that when relevant information is placed in the middle of a long context, major LLMs (including GPT-3.5 and Claude) answer questions significantly less accurately using that information, while content at the beginning (primacy effect) and end (recency effect) is used more reliably. This is believed to relate to the training distribution of Transformer attention mechanisms — models encounter "information at the start or end" samples far more frequently during pretraining and instruction tuning. This has direct implications for RAG system design: not only does retrieval quality matter, but the ordering of retrieved content also significantly affects final answer accuracy.

4. Agents Don't Fail at t=0 — They "Rot Slowly"

The tester ran 10-day continuous sessions, logging latency drift, memory growth, and retrieval degradation every night — a process now ongoing for more than 30 days.

Their conclusion is pointed: "My agents don't fail at the start — they rot slowly." This exposes the fundamental risk of long-running agents: everything looks fine in the short term, but latency gradually accumulates, memory slowly expands, and retrieval quality quietly degrades. Only long-cycle nightly monitoring can catch these slow deteriorations.

5. Only About 10% of Tasks Actually Need the Top-Tier Model

The final finding is about cost. Test data shows that only roughly 10% of agent tasks genuinely require the highest-tier model. There's approximately a 30× cost gap between tiers, but on the remaining 90% of tasks, using a cheaper tier produces no measurable quality loss — provided the runtime actively routes requests rather than defaulting to the most expensive path.

This changed how the tester measures cost: the metric became cost per completed task, not cost per token. For teams trying to control spending on hybrid local+cloud deployments, this is an operationally valuable indicator.

This finding aligns closely with the emerging system design concept of model routing — dynamically assigning requests to models of different sizes at inference time based on task complexity, semantic features, or historical success rates, rather than routing everything through the most powerful model. A typical tiered structure looks like: small local models (1B–3B parameters) for simple classification, summarization, and formatting; mid-size models (7B–13B) for standard tasks requiring some reasoning; and large cloud models reserved only for complex multi-step reasoning, code generation, or tasks requiring broad knowledge. Open-source projects like RouteLLM and LLM-Blender are already working to systematize this approach. The shift to "cost per completed task" as a metric represents a move away from measuring infrastructure efficiency (tokens/second) toward measuring business value delivery efficiency — a better fit for evaluating the real ROI of hybrid deployment architectures.

What This Means for Local Agent Developers

The value of this benchmark isn't in any single number — it's in how it redefines what's worth measuring. Model capability matters, but when an agent needs to run for weeks on a mini PC, the file count threshold in memory, the blast radius of the sandbox, the U-shaped recall of context, the slow rot of long-running performance, and the cost strategy of per-task routing are what actually determine whether the system can work in the real world.

The tester packaged the full experiment into a minimal Kaggle notebook where all of these findings can be reproduced. For developers building local AI agents, this is a checklist worth going through line by line against your own system.