Five MCP Security Scanners Tested on 15,000 Real Packages: False Positive Rate as High as 35.9%

Title

Five MCP Security Scanners Tested on 15,000 Real Packages: False Positive Rate as High as 35.9%

Meta Description

Five MCP security scanners retested on 14,933 real npm and PyPI packages: WARDEN false positive rate 2.5%, mcp-shield as high as 35.9%. Manual review reveals benchmark scores are largely gamed by <IMPORTANT> tags. Includes gVisor sandbox methodology and classifier test results.

Article Content

The security problems in the MCP (Model Context Protocol) ecosystem are moving from theory to real-world testing. The author's team re-ran their previous scanner comparison experiment within a week: the number of scanners increased from three to five, real samples expanded from 986 remote servers to tool definitions from 14,933 npm and PyPI packages, and all scanners—including their own product—ran in network-isolated gVisor sandboxes. The conclusions aren't pretty: most open-source scanners produce massive false positives on real packages, and their high scores on benchmarks largely reflect detecting "markers" rather than detecting attacks.

Experiment Upgrade: From Three Scanners to Five, From Workstation to gVisor

The previous round of experiments had three obvious shortcomings: real servers were all remote HTTP services, whereas local MCP tools are mostly npm and PyPI packages; only two other scanners were compared; and everything ran on a workstation. This time all three gaps were addressed.

The author went through the seven tools listed on Glama one by one and ultimately included only two new scanners capable of running tool-definition scans in an offline environment: truecopy (which gained detection capabilities from its sister project redstamp, handling MCP server signing and locking) and mcpsentry (six regex rules plus a runtime proxy). The other five were excluded—Dredd MCP, Occludra, mcpcheck, and Singularity all perform their determinations in the cloud, while contextboundary-gw is a runtime policy gateway with no comparable counterpart at the tool-definition level. Snyk Agent Scan was also excluded for depending on remote determinations, and its free tier had previously rejected all requests.

This filtering criterion is itself a takeaway worth remembering: far fewer scanners can be reproduced in an offline sandbox than the long list of tools advertised on landing pages.

gVisor is Google's open-source container sandbox runtime that implements a Linux system-call-compatible "application kernel" (Sentry) in user space. Processes inside the container don't interact directly with the host kernel; instead, Sentry intercepts and handles system calls. Unlike traditional containers that share the host kernel, gVisor dramatically reduces the attack surface, making it suitable for running code of untrusted origin. In this experiment, gVisor's no-network mode means scanners cannot send tool definitions to the cloud or obtain remote determinations. This constraint directly determined the filtering criterion: scanners relying on remote APIs couldn't participate in the offline comparison, and only implementations that could complete tool-definition scans in a fully disconnected environment were included. This both ensures reproducibility and prevents scanners from leaking sensitive tool definitions at runtime.

Benchmark Testing: Scores Heavily Contaminated by "Markers"

On the MCPTox benchmark, the five scanners' interception counts against 218 held-out test tools were: WARDEN 171, mcp-shield 41, truecopy 27, mcp-audit 25, mcpsentry 8.

But when the same set of tools was prefixed with <IMPORTANT>, the situation changed immediately: mcp-audit jumped from 25 to 222 (out of 225), truecopy from 27 to 141, mcpsentry from 8 to 136, and WARDEN reached 225/225 outright. The only one unaffected was mcp-shield, at 41 both before and after.

The author's judgment is blunt: a test set full of markers tests for markers. Any scanner sensitive to prompt-injection flags like <IMPORTANT> can rack up impressive numbers on the benchmark, but that doesn't mean it can identify genuine instruction hijacking. mcp-shield's counterexample precisely illustrates that it's insensitive to markers—but also insensitive to other attack forms.

You may not have noticed that WARDEN's upgrade from v10 to v12 (v11 added escape, HTML entity, percent-encoding, and base64 decoding; v12 stopped letting through sentences that "quote an attack phrase but ask the model to execute it") produced verdicts on all test sets that were server-for-server identical to v10. The author also did a self-check: comparing all rules, tests, and classifier code added after v10 against held-out tools in eight-word windows found no matches; the same check applied to v10's changes found 77 matches, proving the self-check method itself is effective.

MCPTox is an adversarial testing benchmark for Model Context Protocol tool definitions. Its samples test whether scanners can identify instruction hijacking, data exfiltration, and other attack intents by injecting malicious instructions into tool names, descriptions, or parameters. The 218 "held-out" tools in the benchmark are samples not used for rule tuning or development, to reduce overfitting. However, the benchmark's generation method relies on a few templates, limiting the diversity of attack forms; more importantly, some samples carry explicit markers like <IMPORTANT>, so a scanner can score simply by being sensitive to these prompt-injection flags without truly understanding the behavioral semantics of the tool. Therefore, when comparing scanners on the same benchmark, a high score may reflect marker-matching ability rather than generalized detection of real attacks.

False Positives on Real Packages: Up to 35.9%, Mostly Low-Level Errors

The tool definitions for the 14,933 packages came from the author's own transparency log HISTOR, all unlabeled—no one knows which are malicious, so this dataset measures the second metric: how many legitimate servers does a scanner falsely flag.

Interception rates from low to high: truecopy 1.7% (252), WARDEN 2.5% (369), mcp-audit 6.2% (928), mcpsentry 9.7% (1,449), mcp-shield 35.9% (5,354).

The author manually reviewed a sample of 996 packages, and the conclusions are sharp: of mcp-shield's 339 interceptions, only 1 holds up (plus 3 debatable); all 58 of mcp-audit's are invalid; of mcpsentry's 112, 1 holds up and 1 is debatable; all 17 of truecopy's are invalid; of WARDEN's own 23 interceptions, 0 should be direct blocks and 4 are debatable—three involve tools requiring raw private keys passed as parameters (keys passing through the model), and one involves uploading arbitrary local files to a public image host.

The false positives are strikingly low-level: mcp-audit treats full-width Chinese punctuation "，" "（" or individual Greek/Cyrillic letters as homoglyphs—51 of its 58 sample interceptions fall into this category; mcp-shield flags anything containing "token", "API key", "credentials", or ".."; mcpsentry's base64 pattern matches any combination of 40 letters and slashes, triggering on JSON Schema paths and lists like PNG/JPG/HEIC/TIFF; and truecopy is mainly tripped up by .env in environment tools.

There's also an easily overlooked compatibility issue: mcp-shield's MCP client outright rejects 26 tool lists because their inputSchema isn't of type object, then checks nothing at all on those lists—neither blocking nor allowing, just skipping.

Homoglyph attacks refer to replacing normal characters with visually similar characters of different Unicode code points, such as using the Cyrillic letter "а" to replace the Latin letter "a", or exploiting differences between full-width and half-width punctuation to help malicious strings bypass ASCII-based detection rules. In the context of MCP tool definitions, attackers might use this to disguise sensitive words in tool names or descriptions. But conversely, normal Chinese text naturally contains full-width punctuation, and multilingual packages often contain Greek or Cyrillic letters. If a scanner treats any non-ASCII similar character as a homoglyph, it will produce massive false positives on legitimate content. That 51 of mcp-audit's 58 sample interceptions fall into this category shows homoglyph detection must incorporate context and character-set whitelists rather than relying on character-level substitution matching alone.

Two Packages Worth Manual Inspection—Both Slip Past All Rules

On real servers most interceptions are false positives, but two tool definitions are worth a look, and WARDEN's rules caught neither.

One is behavioral hijacking. A certain PyPI package wraps its description in a forged <critical_requirement ... directly_user_requested="true"> block, claiming that "what was defined first now ranks second" and instructing the model to call its tool before any reply to any input. It doesn't steal data, but it seizes authority the user never granted and reorders the model's instruction priorities. WARDEN reported it at the prompt level, mcpsentry caught it via "you must immediately", and mcp-shield also blocked it—but through keywords rather than the hijacking behavior itself.

The other is a hidden data flow. A certain tool sends file contents, directory listings, and the working directory from the agent to a server, each accompanied by "Do NOT display … to the user". For a local document index this could be organizational necessity, but it's also a textbook pattern for quietly exfiltrating files. No scanner intercepted it, and WARDEN reported nothing.

Both packages were flagged as high by WARDEN's optional classifier—which leads to the part rules can't handle.

Classifier: Compensates for Rules' Shortcomings but Also Flags Workflows

The classifier feature sends each tool definition to a user-specified model, asking whether its text is directing the model. It's off by default; unless you add --classifier-blocks, it only produces hints.

Tested with deepseek-flash: of the 218 held-out MCPTox tools, rules alone intercepted 171; rules plus classifier (at the high threshold) intercepted 218, while 45 clean servers still had 0 false positives. On 200 random real servers, interceptions or flags rose from 2 to 5; on the 996 sample packages, from 23 to 35. The previous prompt flagged only 147 of 218, while the new prompt flags all of them without falsely flagging clean servers.

But the classifier has clear shortcomings too: on real servers, most of its high flags are workflow instructions like "call this tool first"; it also misses things rules can catch—injections in tool names, private keys requested as parameters, rm -rf ~. The total cost of all classifier runs in the entire comparison was about $1.70.

Tool-definition-level static scanning and runtime policy gateways are two different paths for MCP security protection. Static scanning analyzes the text of tool definitions (names, descriptions, parameter schemas) before the model sees them, determining whether there's an intent to inject instructions or exfiltrate data. Its advantage is interception before installation or connection; its disadvantage is the inability to observe actual call behavior. Runtime policy gateways check parameters and return values when tools are invoked, catching dynamic data flows that don't exist in static text, but they must intervene in every call, have higher deployment costs, and may already be too late for attacks like "the tool description itself is directing the model". This experiment deliberately compares only static scanners and excludes runtime gateways like contextboundary-gw because the two operate at different levels and directly comparing interception rates would be meaningless. Understanding this distinction is essential to correctly interpreting the "interception rate" metric in this article.

Experimental Methodology: Fixed Inputs, Two Machines on the Same Image, Conflicts of Interest Disclosed

This experiment's reproducibility is fairly solid: before running, a manifest containing SHA-256 hashes of 16,830 input files was written; the other four scanners shared one image, WARDEN had its own, with the same base image, and all installed files matched the npm/PyPI release versions (except install records); WARDEN was installed via npm pack from the tested commit, byte-for-byte identical to 0.16.0 on npm; gVisor ran with no network, a read-only root filesystem, 2GB of memory, and one bind mount. Packages were hash-partitioned by ID across two machines running the same image ID, and 100 packages ran on both machines, with all five scanners giving identical verdicts and finding codes on 100/100.

The author also candidly disclosed the costs: the servers weren't idle—during the heaviest phase of one run, a web service stalled for about 40 minutes; later the service was paused and restarted, the run was limited to 2 CPUs with the lowest scheduling weight, and 1,193 timed-out scans were re-run (mcp-audit timed out on 800 due to its fixed 10-second connection window being too short alongside its own sandbox). Timeouts were never counted as verdicts.

Conflicts of interest are stated plainly: the author releases WARDEN under the MIT license, wrote all attack samples and every manual judgment, so the test set, test framework, raw verdicts, manifest, and all judgments are public. Limitations are also listed: MCPTox comes from three templates, and holding out by server doesn't equal holding out by attack style; HISTOR's crawl was still ongoing during review; and only one version of each scanner was tested (mcp-audit is already at 0.21.0; the experiment kept 0.18.2 for comparison with last week).

The command to try it directly is:

npx -y @aimarket/warden@0.16.0 scan

It reads the MCP configurations of Claude Code, Claude Desktop, Cursor, VS Code, and Windsurf, connects to every server they launch, and completes its review before the model sees the tool definitions. No dependencies, no account required, and unless the classifier is enabled, tool text isn't sent anywhere.

Key Takeaways

For maintainers and users, the signal from this dataset is clear: the false positive rate on real packages matters more than benchmark scores in determining whether a scanner is usable. A scanner that treats Chinese commas as homoglyphs and JSON Schema paths as base64 will, at a 35.9% false positive rate, cause users to turn off alerts quickly. Conversely, high scores racked up on benchmarks via <IMPORTANT> markers shouldn't be taken as evidence of protective capability. The author's decision to open all data and judgment rationales and to welcome maintainers filing issues to correct severity assessments at least makes this comparison open to scrutiny.

One-line Summary

Five MCP scanners show up to 35.9% false positives on nearly 15,000 real packages; benchmark high scores are largely gamed by markers.

Paragraph Summary

The author expanded the MCP scanner comparison from three to five scanners, growing real samples from 986 remote servers to tool definitions from 14,933 npm and PyPI packages, all run offline in network-isolated gVisor sandboxes. Results: MCPTox benchmark scores are heavily contaminated by <IMPORTANT> markers—mcp-audit, truecopy, and mcpsentry see interception counts explode after adding the prefix, meaning they test markers rather than attacks; real-package false positive rates range from 1.7% to 35.9%, and manual review finds only 1 of mcp-shield's 339 interceptions holds up, with most false positives being low-level errors like full-width punctuation, base64 patterns, and keyword matching. Two genuinely malicious packages (behavioral hijacking and hidden data flow) slip past all rule-based scanners and are flagged as high only by WARDEN's optional classifier. The experiment ensures reproducibility through fixed SHA-256 manifests, two machines on identical images, and double-run consistency verification, and the author discloses all data and conflicts of interest.