Daily AI Fresh News · Oct 18 Morning Edition: MCP Scanner False Positives and the AI Evaluation Business

Lots of new moves in the AI world today. One experiment completely stripped the MCP scanners down to their underwear, and there's a pretty interesting topic too—websites are starting to leave notes for AI agents.

Today we're mainly covering these few things: first, backend-less crypto payments, second, MCP scanners falling flat on their face, third, agent scheduling primitives.

Fourth, websites leaving notes for agents, fifth, the Gemini 4 Argon update, sixth, the business of scoring AI. Yep, that's six things, let's go one by one.

Yeah, all of them are worth talking about.

Let's start with the first one, this is pretty interesting. There's an article on Dev.to about how to verify USDT payments without a backend. Li Bo, guess how they did it?

No backend? Then most likely it's talking directly to on-chain nodes. I'm guessing the browser calls a public RPC.

Exactly, a static site calling a public node in the browser, three eth_call reads of the store wallet's BEP-20 USDT balance, using the balance difference to determine if payment arrived.

I'm very familiar with this approach—essentially using the chain as a database. Read a baseline first, then poll every twenty seconds, with a zero point one percent tolerance, sixty minutes expiry.

Saves the database, saves the event indexer, cost is basically zero. But there's a catch: with concurrent buyers, attribution goes wrong—you don't know who paid.

Right, so the 'paid' shown in the browser can only be a convenience hint. For real confirmation, the store owner still has to manually check on BscScan.

And it only supports USDT on BNB Smart Chain—send on the wrong network and the money's gone forever. That's pretty scary.

So it only suits low-frequency, low-concurrency service transactions, delivery cycle one to two days, no refunds. Pick the right scenario and it's pretty sweet.

Second one, this next experiment is something else. Someone redid the MCP scanner comparison, scaling the sample from nine hundred-something to over fourteen thousand tool definitions from npm and PyPI packages.

I've seen this—the key point is it all ran offline in an air-gapped gVisor sandbox, and they added two more scanners, five in total for the comparison.

And the results?

The results were a slap in the face. The MCPTox benchmark scores were severely contaminated by the IMPORTANT marker—once you add the prefix, the interception counts for several scanners just exploded. They weren't measuring attacks at all, they were measuring that marker.

That's a bit awkward. So how did they do on real packages?

False positive rates ranged from one point seven percent all the way to thirty-five percent—huge difference. Manual review found that of mcp-shield's three hundred-plus interceptions, only one actually held up.

Three hundred-plus interceptions for one real hit—that precision is kind of absurd. So what were the false positives?

All low-level mistakes—full-width punctuation, base64 patterns, keyword matching. Worse, two real malicious packages—one behavior hijacking, one hidden data flow—all the rule-based scanners missed them.

Only the optional WARDEN classifier flagged them as high. So the takeaway is rule-based scanning is basically useless?

At least it shows that treating benchmark scores as proof of security capability is very dangerous. That said, the experiment was solid—fixed SHA-256 manifests, dual machines with the same image, double runs to verify consistency, all data public, and they disclosed conflicts of interest.

Third, let's talk about agent scheduling. There's an article starting from Akiflow's drag-and-drop time blocks, saying it's built on three layers: task metadata, calendar sync, and conflict resolution.

Good angle. For agents to autonomously schedule execution windows, this is exactly what's missing. Everyone's always staring at model capability—nobody looks at the scheduling foundation.

The article groups the hard problems into three categories: eventual consistency across multiple calendar APIs, rate limits and concurrent booking conflicts, and constraint solving for time windows.

That last one is the hard bone—essentially a constraint satisfaction problem. Greedy works okay, ILP finds the optimum, but you have to trade off latency and results.

It also gives a four-layer architecture—scheduling engine, task queue and metadata, calendar sync layer, execution runtime—plus failure modes and an observability table.

It even covers security isolation details like OAuth and rate limit pooling, and includes minimal scheduler code. Anyone building agent products should really read it.

Alright, that's the news for now. Next let's dive into two topics: websites leaving notes for agents, and the business of scoring AI.

First one—recently a topic went pretty viral, saying websites are starting to leave notes for AI agents, and one of them just says 'meow.'

I've seen this—it's essentially an extension of the robots.txt idea. Before it was telling crawlers whether they can scrape; now it's talking directly to agents.

Right, but why is it blowing up now of all times? Think about it—agents actually browse websites, place orders, fill out forms on their own now. Site owners need some channel to speak.

That's not the main point. The main point is this signal shows agents have become visitors websites have to take seriously. Traffic used to be people; now it's mixed with agents.

But here's the problem—if a note is written, will agents actually listen? There's no enforcement power behind this at all, right?

Zero enforcement—it's a gentleman's agreement. That's exactly what people are arguing about on Reddit. One camp says it's the new robots.txt; the other says it's pure self-comfort.

I lean toward the latter. At least with robots.txt, search engine vendors had consensus to follow it. On the agent side there are too many vendors, too scattered—who's going to guarantee it?

But think again—consensus has to start somewhere. Early robots.txt was ignored too, then it became an industry default. Maybe agents will follow the same path.

Fine, I'll reserve judgment on that. But that 'meow' is pretty vivid—like the site owner is being cute with the machine.

That's actually a kind of probing—expressing a serious demand in a joking way: please stop scraping recklessly. I think the buzz shows people are really anxious about agent behavior boundaries.

So to sum up: leaving notes for agents isn't technically sophisticated, but it marks that site owners are starting to actively negotiate with agents. That shift in posture is the key.

Second topic—there's a video on Bilibili discussing the business of scoring AI, with a catchy title: 29 people making a hundred million dollars a year.

My first reaction to that number was shock; my second was 'makes sense.' Benchmarks are the standard weights and measures of the AI era—whoever holds the ruler holds the say.

But think about it—29 people doing evaluations, a hundred million a year. Isn't the business model a bit delicate? Who are the buyers?

Buyers are basically model vendors and investors. Vendors need leaderboards to prove they're strong; investors need leaderboards to decide who to fund. The demand is real.

But isn't that contradictory? The vendors are both the evaluated and the payers—how do you guarantee independence?

That's exactly what programmers are fighting over most fiercely on HN. One camp says as long as the evaluation methodology is open and transparent it's fine; the other says whoever pays you, you speak for—can't escape it.

I'm with the latter. Look at that MCP scanner experiment—benchmark scores can be contaminated by a single marker like that. The leaderboard waters are much deeper than you'd think.

I'd give that analogy full marks. But then again, you can't do without benchmarks—you can't have every user run their own evaluation before choosing a model.

So the question isn't whether to have benchmarks, but how to make them trustworthy. Things like third-party audits, cross-verification across multiple leaderboards.

Right, and I predict more open-source, community-built evaluation systems will emerge, and the credibility of commercial leaderboards will keep being challenged. That's a fairly confident call from me.

But that Gemini 4 Argon update video also got high engagement—five hundred-plus. Seems people still care more about the models themselves.

That's normal. Model updates are directly usable; the evaluation business is behind-the-scenes infrastructure. Naturally different levels of buzz. But infrastructure determines the long-term landscape.

So the conclusion: AI evaluation has gone from an academic trifle to a hundred-million-dollar business, but its independence—like the MCP scanners—needs serious scrutiny.

Well summed up. Today's topics actually form one line: agents are getting deeper into the real world, while the trust and measurement systems around them haven't caught up.

Well said. That's it for today—lots to digest, take your time. See you at noon!