Forecall
A research team focused on AI tools and the MCP ecosystem that performs structured scoring of MCP tool description quality and end-to-end model testing.
Timeline (last 90 days)
The Forecall team conducted structured scoring of the descriptions of 55 public MCP servers, totaling 1,274 tools; the average scores of the 55 servers ranged from 19.9 to 82, with a median of only 44.
Among the 1,274 MCP tools, 497 tool descriptions scored below 40, and only 18 reached 80 or above.
885 tools (69%) never stated their usage scenarios in their descriptions.
Among the 156 directly extracted tools, 87 (56%) had neither an output schema nor a description of what they return.
Of the 55 MCP servers, 18 had at least one pair of nearly identical tool descriptions, totaling 129 pairs, with Notion (57 pairs among 24 tools) and Paper Search (29 pairs among 25 tools) contributing the vast majority of the duplication.
Forecall, on 88 tools across 4 servers (Memory, Notion, Playwright, Chrome DevTools), generated 8 single-step requests per tool and tested model selection accuracy with Claude Sonnet 5.5 and GPT-6.1 Sol respectively, with accuracy across all servers ranging from 91% to 99%.
The Spearman correlation coefficient between tool description scores and correct model selection was only 0.04, with almost no statistical association.
Tools scoring below 40 were correctly selected 95.5% of the time, while tools scoring 60 to 79 instead reached 100%.
After mixing 25 to 55 tools from other servers into the candidate set, not a single request was misdirected to another server's tools, and accuracy remained between 91% and 98%.
When frontier models face a request with an unspecified page number, in 77% to 87% of cases they first call list_pages before performing the operation; for browser_fill_form, the model first obtains a page snapshot.
3 more timeline events
All Facts (13)
The Forecall team conducted structured scoring of the descriptions of 55 public MCP servers, totaling 1,274 tools; the average scores of the 55 servers ranged from 19.9 to 82, with a median of only 44.
50%UnverifiedAmong the 1,274 MCP tools, 497 tool descriptions scored below 40, and only 18 reached 80 or above.
50%Unverified885 tools (69%) never stated their usage scenarios in their descriptions.
50%UnverifiedAmong the 156 directly extracted tools, 87 (56%) had neither an output schema nor a description of what they return.
50%UnverifiedOf the 55 MCP servers, 18 had at least one pair of nearly identical tool descriptions, totaling 129 pairs, with Notion (57 pairs among 24 tools) and Paper Search (29 pairs among 25 tools) contributing the vast majority of the duplication.
50%UnverifiedForecall, on 88 tools across 4 servers (Memory, Notion, Playwright, Chrome DevTools), generated 8 single-step requests per tool and tested model selection accuracy with Claude Sonnet 5.5 and GPT-6.1 Sol respectively, with accuracy across all servers ranging from 91% to 99%.
50%UnverifiedThe Spearman correlation coefficient between tool description scores and correct model selection was only 0.04, with almost no statistical association.
50%UnverifiedTools scoring below 40 were correctly selected 95.5% of the time, while tools scoring 60 to 79 instead reached 100%.
50%UnverifiedAfter mixing 25 to 55 tools from other servers into the candidate set, not a single request was misdirected to another server's tools, and accuracy remained between 91% and 98%.
50%UnverifiedWhen frontier models face a request with an unspecified page number, in 77% to 87% of cases they first call list_pages before performing the operation; for browser_fill_form, the model first obtains a page snapshot.
50%UnverifiedIn earlier tests, a rewrite that raised a tool's score from 31 to 74 instead made both models' selection performance worse.
50%UnverifiedForecall's linter is already open-sourced on GitHub, offering an online check (forecall.dev/en/lint) and a local command-line tool (npx forecall dump and npx forecall lint), and local use does not send data externally.
50%UnverifiedFor Forecall users on the Pro or Team plans, the evaluation feature can report metrics such as model selection accuracy, upfront lookup rate, parameter validity, and erroneous calls.
50%