Is context-mode's 98% Savings Overstated? The Real Challenge Is the Model Writing Scripts

Title

Is context-mode's 98% Savings Overstated? The Real Challenge Is the Model Writing Scripts

Meta Description

context-mode claims 98% context savings, but independent testing reveals the truth: the number isn't fake, but it only measures the ideal case of hand-written scripts. In real scenarios, the cost of models writing scripts, the risk of parsing errors, and flaws in index retrieval are what really determine AI agent efficiency.

Article Content

An open-source tool called context-mode has recently sparked heated discussion in developer communities. It claims to save 98% of the context window usage for AI coding agents—a number that even made the front page of Hacker News, garnering 570 points. But an independent reproduction test revealed an interesting fact: the 98% figure itself isn't fabricated, but it only measures the upper limit under ideal conditions—the real challenge lies in getting the model to write correct extraction scripts.

What Is context-mode: Sound Approach but Questionable Implementation

Developed by Mert Koseoğlu, context-mode is a combination of an MCP server and hooks for over a dozen coding agents including Claude Code, Codex, Cursor, and Gemini CLI. Its core idea is quite clever: instead of having the tool dump a 56KB page snapshot directly into the conversation, it has the agent write a small script that context-mode executes in a subprocess, returning only the script's printed output to the model. Large outputs are indexed into a local SQLite full-text search database that the model can query later. The project also offers session memory that survives context compaction.

The project uses the Elastic License 2.0 source-available license, having initially been released under MIT before switching in March. It trended on GitHub, gaining 904 stars in a single day and approaching 30,000 total stars.

MCP (Model Context Protocol) is a protocol proposed and open-sourced by Anthropic to standardize how AI models connect to external tools and data sources. An MCP server is essentially a separate process that exposes a set of tools (such as reading/writing files, executing commands, querying databases) to the model, which invokes them through a structured protocol and receives results. Coding agents like Claude Code and Cursor all support connecting to MCP servers. context-mode works precisely as an MCP server plus hooks: when the agent calls its tools, execution is delegated to a subprocess, and only the condensed result is returned to the model. The significance of this architecture is that tool outputs don't all need to enter the conversation context, providing a technical opening for context compression.

What Exactly Does 98% Measure?

This figure comes from the project's bundled tests/ecosystem-benchmark.ts. It contains 14 scenarios, each pairing a saved tool output (Context7 documentation, a Playwright-captured Hacker News snapshot, 20 React issues, a test run, nginx logs, etc.) with a summary script hand-written for that specific file. The raw size is the file itself, and the context size is what the script prints.

In other words, 98% compares file size against the output of a script written by someone who has already seen the file. This does fairly measure how small a good summary can be, but it doesn't show the agent actually working. The tester ran the entire benchmark with the project's own executor, and every line matched BENCHMARK.md down to the byte.

Context Window is the maximum number of tokens a language model can process in a single inference, with conversation history, tool calls, and their return values all consuming this budget. Coding agents repeatedly read files, execute commands, and check logs while working—outputs that can easily fill the window, forcing the system to compress or discard earlier content. "Saving context" essentially means reducing the volume of tool output entering the window, leaving more room for reasoning and task state. But saving presupposes that the model can trust the trimmed summary—it gives up the ability to directly inspect the raw data. context-mode's 98% figure measures compression ratio on this dimension without counting the token cost of generating the summary script itself, so when evaluating context cost, the "code the model writes for compression" must also be included in the total overhead.

The Overlooked Context Cost

In real use, that script is written by the model itself, and the script is itself part of the context. It enters the conversation as a tool call. The current server also echoes it back: every ctx_execute_file reply begins with the file path and a code block of up to 2,000 characters before the output. The benchmark counts neither.

Recounting the same 14 scenarios under three measures: raw tool output is 322,419 bytes, stdout alone is 5,517 bytes (98.3% savings), adding the echoed path and code brings it to 16,916 bytes (94.8% savings), and adding the code in the call itself reaches 27,504 bytes (91.5% savings). These scripts total 10,588 bytes—nearly twice what they print. For the smallest one, a 400-byte network request list, a single round trip costs 2,414 bytes, six times the content it was supposed to compress.

91.5% is still a substantial saving. The problem isn't that the number is fake, but that it's a best case—measured after the hardest part is already done.

The Script Is the Real Difficulty

Some samples in the benchmark file clearly come from real captures: the React issues match GitHub item by item. But the git log sample doesn't look like real data—30 of 31 days had exactly five commits, and four of the five authors are Alice Johnson, Bob Martinez, Charlie Kim, and Diana Chen, every name being two words.

This matters because the script's split(" ")[4] assumes the fifth word on each line is the commit type, which only holds if author names happen to be two words. The tester ran the exact same script on context-mode's own commit history (2,234 commits, 203KB). The summary went from 107 bytes in the benchmark to 7,071 bytes, still a 96.5% saving, but the first entries were wrong: the output showed the most common type as "update" with 552 occurrences—when in fact 550 commits came from github-actions[bot], a one-word name, and the script read "update" from "ci: update install stats". Three-part names made the script treat the author's surname as the commit type. In total, 679 commits (30%) had author names that weren't two words.

No error was raised. The output looked like a summary. The agent reading it had no reason to doubt, because the 203KB of raw data that could have exposed the error never entered its context. That's the trade-off: the saving comes from the model not seeing the data, and the risk comes from the same place.

Safety Net Tested: Index Retrieval Isn't Ideal Either

context-mode's answer to information loss is indexing. When you pass an intent and the output exceeds 5KB, it chunks the output into SQLite FTS5 and replies only with the best-matching passages. The tester used gh issue list --json to fetch 100 open React issues (303KB), built the index with the project's own storage, and constructed the reply the way intentSearch does in server.ts, with the query intent "ViewTransition bugs".

The reply was 1,667 bytes—99.4% savings. But eight of the 100 issues mentioned ViewTransition, and none of their numbers appeared in the reply. Because gh prints JSON as a single line, the chunker splits by bytes, so all five matching titles were "Output", each preview being a 120-character slice of raw JSON, mostly from the middle of issue bodies. The reply does prompt the model to call ctx_search for full passages, and a careful model would do so—but that's another round trip, and the first reply itself barely answers anything.

SQLite FTS5 is SQLite's built-in full-text search extension, which supports building inverted indexes over text and executing keyword-matching queries. context-mode uses it to store large tool outputs in chunks, so the model can later retrieve relevant passages via query intent without loading the full text at once. FTS5's retrieval quality depends heavily on the chunking strategy: if split at fixed byte boundaries, a single chunk may span multiple semantic units, preview text loses structure, and query hit rates decline. For compact single-line formats like JSON, byte-based chunking is especially prone to splitting titles, numbers, and bodies that should appear together, producing retrieval results that look relevant but can't be used directly. This shows that for an index to work as a "safety net", the chunking method must match the data format.

Practical Advice

The tester's conclusion: adopt the approach, but don't rely on this package. When command output is large, have the agent write it to a file and run a script to process it, while requiring the script to print a few raw sample lines alongside the counts, so parsing errors expose themselves. Evaluate such tools by task rather than by bytes: did the agent get the right answer, at what cost, counting the code it wrote.

The test's limitations are also clear: context-mode was never actually run inside an agent, so it can't determine the probability that a real model writes the script correctly on the first try, nor assess session memory after compaction. The project's GitHub issue tracker is closed, so user feedback can't be reviewed.

This raises a question every AI agent user should consider: when your agent summarizes output you've never seen, what do you check before trusting it?

One-line Summary

context-mode's 98% savings figure is real but represents an ideal upper bound; actual savings are about 91.5%, and script parsing is error-prone.

Paragraph Summary

An independent reproduction test shows that the open-source tool context-mode's claimed 98% context savings isn't fabricated, but it comes from an ideal benchmark with hand-written summary scripts for each file, and doesn't count the context cost of the model generating scripts. Actual savings are about 91.5%, with negative optimization for small files. The test also found that the benchmark's git log sample appears to be synthetic data, and the script relies on the assumption that author names are two words—on real commit history, 30% were misclassified with no errors raised. The SQLite index retrieval safety net also fails to reliably recover key information when chunking JSON by bytes. The author recommends adopting the "have the agent write scripts and print sample lines" pattern, and evaluating such tools by task completion quality rather than byte savings.