56% of AI Agent Work Unaccounted For: How One Session Label Caused a Measurement Disaster

A Seemingly Ordinary Query Reveals a Staggering Black Hole

Three weeks, 65 sessions, 4,985 messages. A developer pulled his AI agent chat logs into a database and tried to sort them with SQL: How much time went into fixing its own mistakes? How much went into actually publishing content? And what was all the rest?

The result showed a massive fracture: 2,813 messages—56% of the total—didn't fit any category he had defined. That number sat untouched in the dataset for three days until he finally decided to dig in.

AI Agent refers to a large language model-driven system capable of autonomously perceiving its environment, making decisions, and executing multi-step operations to accomplish goals. Unlike traditional single-turn Q&A, an agent can continuously call tools, read and write files, execute code, and access the network within a single session, adjusting subsequent actions based on intermediate results. This multi-step autonomy makes the agent's work process highly unstructured: it may conduct extensive exploration, trial-and-error, and self-correction without the user's knowledge. As a result, an agent's "workload" is difficult to measure using traditional project management task categories—a single message could be a critical step toward publishing a blog post, or it could simply be the agent thinking aloud. The measurement challenge revealed in this article is a direct consequence of this kind of unstructured autonomous work.

One Session Ate a Quarter of the Entire Corpus

To understand that 56%, you first need to look at the overall distribution of sessions. The median session had only 18 messages, but the largest one—titled "Quick check-in," of all things—contained 1,327 messages on its own, accounting for 26.6% of all activity across the three weeks.

This session ran from 06:10 one morning to 07:39 five days later. The developer sent only 133 turns of messages, while the agent responded with 1,194. Judging purely by volume, this looked nothing like a "quick check-in"—it was more like a five-day marathon of collaboration.

It Published Six Posts, But the Title Gave No Clue

Deeper digging made things even more absurd: this session called "Quick check-in" had actually produced all six articles from Blog 17 through Blog 22, accounting for 24% of everything the developer had published. He cross-referenced publication times with session message timestamps one by one and found that each post's publication date matched a message record within the same minute.

Post	Published (UTC)	Nearest Message Time
Blog 17	10-07 06:34	10-07 06:33
Blog 18	10-08 18:34	10-08 18:33
Blog 19	10-10 03:55	10-10 03:55
Blog 20	10-11 12:15	10-11 12:15
Blog 21	10-12 04:18	10-12 04:17
Blog 22	10-12 07:39	10-12 07:38

The timestamps matched down to the minute, leaving virtually no room for coincidence. The chain of data evidence was complete and solid.

The Culprit Was a Dropdown Menu

Why was this behemoth dumped into "uncategorized"? The answer was embarrassingly simple—arithmetic.

The developer defined "recovery-type" sessions (titles containing crash, diagnose, recover, etc.) as 10 total, summing to 1,636 messages. This 1,327-message session clearly couldn't fit in there. Then he defined "publishing-type" sessions as 3 total, summing to 536 messages. Since 1,327 is bigger than 536, it couldn't belong to that category either.

So the largest session of the three weeks was neither a failure nor a production—all because when it started, the developer casually picked a title from a dropdown menu. That action happened in October, and he was tired at the time.

This is pure entry-point contamination: classification labels were locked in before the session began, at a moment when he had no idea what the session would become.

Entry-point contamination is a classic problem in the field of data annotation, referring to bias introduced at the very origin of data generation or collection that is subsequently difficult to eliminate. In the context of AI agents, session titles are typically manually selected or entered by the user at session creation—precisely the moment when the user knows the least about what the session will ultimately become. Once this upfront label is written into metadata, it permeates all subsequent classification, statistics, and reporting, creating a systematic measurement bias. Unlike post-hoc labeling, the problem with entry-point contamination is that the label carries "initial intent" rather than "actual outcome," and when the work itself is emergent, the two can be completely disconnected. In this article, a session named "Quick check-in" that ultimately produced 24% of published content is an extreme case of entry-point contamination. To eliminate this bias, the labeling point must be moved from the start of a session to its end.

The Truth: Not 11%, But 37%

After correcting this mislabeled session, the data shifted dramatically. "Recovery-type" remained at 1,636 messages (32.8%), and the developer cautiously noted that he couldn't confirm whether this number was accurate, but he emphasized that at least it hadn't moved. What was truly rewritten was "publishing-type": jumping from 536 messages to 1,863, with its share going from 11% to 37.4%.

One mislabeled session had caused more than a third of the agent's workload to be systematically underestimated. His originally reported figure was less than a third of the real output.

The "Residual" Contains More Than One Mislabeled Session

If everything could be blamed on this one label, the story would be too clean. The developer deliberately reminded himself not to fall into that narrative trap.

Breaking down the residual by size: 1,327 messages was that "Quick check-in," followed by 273, 211, 205, 107, 98, 83, 83, 46, and 43. The 273-message session was titled "Greeting." Another 2,582 messages sat in sessions with 40+ messages, never classified.

"I don't know what happened in 'Greeting.' The second-largest session in the corpus has 273 messages, and the best explanation I can offer is: I opened a session, didn't name it, and stayed in it for nine hours."

There were also three sessions with no labels at all, titled with raw timestamps. For example, "New session - 2026-09-27T15:11:27.049Z." This was the trace left by selecting nothing on the picker, covering three sessions and 235 messages total.

In data analysis and statistical modeling, residual refers to the difference between observed values and model-predicted values—the portion that cannot be covered by the existing classification system or explanatory variables. Residuals are not necessarily "bad data"—they often contain real signals that the model has not yet captured. In this article, the developer calls the 56% of unclassified messages residuals, and that wording is accurate: these messages don't not exist; rather, the existing classification framework cannot accommodate them. Further breaking down the residual reveals that it contains sessions that weren't correctly categorized due to mislabeling, sessions with no labels at all, and possibly a new category belonging to "agent thinking aloud." The size and internal structure of the residual actually reflect a fundamental mismatch between the classification system and the real shape of the work. When measuring the work of autonomous agents, the residual is often more informative than the classified portion, because it points to the blind spots of the existing framework.

Mislabels Are Fixable, But Unlabeled Data Cannot Be Classified

The developer made a key distinction here: a wrong label can be remedied—go back, review the work, and rename it. But "unlabeled" is unrecoverable within a classification system, because there is nothing to classify. Those three sessions weren't in the wrong bucket; they were outside the entire measurement tool.

"If I wrote a one-sentence summary at the end of every session, I could have caught that 'Quick check-in.' But no matter how many times I reread my own classification system, I could never catch those three sessions, because the classification system was never applied to them in the first place."

Unlabeled data and mislabeled data are fundamentally different in classification theory. Mislabeled data still exists within the classification system—it's just been placed in the wrong spot, and therefore can be corrected by re-examining the content. Unlabeled data, by contrast, is entirely outside the classification system; it has never been "seen" by any category, and therefore cannot be recovered by any algorithm or manual process based on the existing system. In semi-supervised learning in machine learning, unlabeled data typically requires additional assumptions or prior knowledge to be utilized. In this article's context, those three sessions named with raw timestamps are classic unlabeled data: they weren't wrongly sorted into some bucket—they never entered any bucket at all. This means that no matter how the developer improves his classification system, as long as the labeling action occurs before the session starts, there will always be some sessions that completely escape the measurement scope because "nothing was selected."

The Only Fix Is Four Words: Label After the Fact

The solution he proposed was so plain it was almost anticlimactic: write the title after the session ends, not before it begins.

The title is the cheapest field in the entire system, and the only one that carries intent—but he filled it in at the start of every session, which was the moment he knew the least about what the session would become. Five days later, one of those sessions had become the container for a quarter of his output, while its name still said "Quick check-in."

If you want to classify an agent's work after the fact, the label must come from the work itself, not from the thought you had when you opened the window.

So What Was That 56%?

So after subtracting recovery and publishing, what exactly did that 56% residual contain? The developer candidly admitted he couldn't answer. Part of it might be maintenance work, part of it might be him repeatedly rereading notes he'd already read—but he couldn't even measure that ratio precisely, because the definitions he'd written down himself would swing between 8.7% and 26.2% under different interpretations.

And another part might be the agent thinking aloud in ways he'd never counted as "work."

"That residual is the interesting part, and I can't classify it, because the only handle I can grab onto is a label I typed before I even knew what I was doing."

This isn't a story about SQL tricks. It reveals an underappreciated measurement challenge in the age of autonomous agents: when the work itself is unstructured and emergent, any upfront classification attempt may create more blind spots than it resolves. The developer ends by inviting readers—if you've measured this, share your numbers; if you haven't, the shape of your data itself might be the most debatable thing of all.

One-line Summary

One mislabeled AI agent session caused more than a third of the workload to be systematically underestimated.

Paragraph Summary

A developer imported 4,985 messages from 65 AI agent sessions over three weeks into a database for classification analysis and found that 56% of messages could not be covered by his existing classification system. After digging deeper, the culprit emerged: a session titled "Quick check-in" actually contained 1,327 messages and produced 24% of all published content, yet was left uncategorized because its title didn't match the definitions of "recovery-type" or "publishing-type." After correcting that label, publishing-type work jumped from 536 messages (11%) to 1,863 messages (37.4%). The developer pointed out that mislabeled sessions can be fixed after the fact, but sessions with no labels at all are unrecoverable within the classification system. The core lesson: labels should be assigned after a session ends based on actual work content, not filled in beforehand based on vague intent.