Webpage to JSON Tool: Structured Data with Evidence for AI Agents

LLM Agents are great at reasoning but poor at reading messy HTML. The traditional approach is to write custom scrapers for each website, but as soon as the site is redesigned, the scraper breaks. A tool called Webpage to JSON for AI Agents on the Apify platform attempts to solve this problem with a different approach: input a URL and a natural-language description of the fields you need, and it returns directly a JSON with type declarations, where every value comes with a reference to the original text on the page, making it easy to verify that the data actually exists.

Core mechanism: natural-language instructions + evidence chain

The tool's input is very simple: a list of URLs plus a single instruction written in plain English. Taking the official real-run example, the input instruction is "extract the book title, price (number), currency code, and whether it is in stock", and the target page is a book detail page on books.toscrape.com.

In the output, the data field provides the structured data: title "A Light in the Attic", price 51.77, currency code GBP, stock status true. More importantly, the evidence field records the original source of each value on the page—for example, the title corresponds to "# A Light in the Attic" and the price to "£51.77". There is also an unverifiedFields array that flags fields for which no textual basis could be found.

This design targets the common "hallucination" problem when LLMs extract data—the model may fabricate a plausible-looking price or stock status out of thin air. By mandating evidence references, users can quickly judge which values are trustworthy and which need manual review.

From single extraction to batch list mode

The tool offers several practical options for different scenarios. If you need a precise output structure, you can pass extractionSchema to specify a JSON Schema; for pages containing multiple records, such as product listing pages or job boards, set listMode: true to extract multiple records at once.

The requireEvidence option is enabled by default, meaning any value lacking a textual reference will be placed in unverifiedFields. For scenarios with high data-quality requirements, this default behavior acts as an automatic quality-check gate.

JSON Schema is an open standard (IETF draft specification) for describing and validating the structure of JSON data. Through keywords such as type, properties, required, and items, it declares which fields a JSON object should have, the type of each field, and which fields are required. In LLM structured-extraction scenarios, passing a JSON Schema to the model as an output constraint can significantly reduce issues such as missing fields, type errors, and extra fields. The extractionSchema option leverages this mechanism to let users hand the business-side field contract directly to the extraction service, thereby obtaining stable output aligned with downstream systems (database tables, API contracts) without relying on natural-language descriptions and "luck".

Integration: dual channels via API and MCP

The tool's invocation is developer-friendly. With Apify's Python client, a few lines of code are enough: pass the token to create a client, call the corresponding Actor, pass in the URL and instruction, then iterate over the dataset and print the data field of each result.

Even better suited to Agent scenarios is the MCP integration. Add https://mcp.apify.com/?actors=s3nafps/webpage-to-json-for-ai-agents to Claude, Cursor, or any MCP client, and the Agent can treat this tool directly as a callable capability. This means that when building Agents that need to read web page information, you no longer have to write separate parsing logic for each data source.

MCP (Model Context Protocol) is a standardized protocol proposed and open-sourced by Anthropic for letting LLM applications connect to external tools and data sources in a unified way. It uses a client–server architecture: host programs such as Claude Desktop and Cursor act as clients, while tool providers act as servers, exchanging tool lists, invocation requests, and results via JSON-RPC. Its value lies in decoupling—the same tool server does not need a separately developed plugin for each host; once integrated, it can be reused by multiple Agents. The mcp.apify.com/?actors=... given in the article is a kind of remote MCP server address, mounting a specific Actor on demand via URL parameters, enabling the Agent to dynamically acquire the "webpage to JSON" capability at runtime.

Pricing and scope of applicability

In terms of cost, LLM costs are included, and each page from which data is successfully extracted is charged $0.03; pages where no data is found are not charged. This result-based billing model is fairly friendly for trial use.

It should be noted that such tools cannot fully replace custom scrapers. For pages with extremely complex structures, requiring login sessions, or needing dynamic rendering, results may suffer; moreover, the phrasing of natural-language instructions also affects the stability of extraction results. It is better suited to rapid validation, low-frequency data collection, and data-verification scenarios that need an evidence chain. For high-frequency, large-scale production-grade scraping, custom solutions still have advantages in cost and controllability.

Dynamically rendered pages refer to pages whose content is generated by JavaScript executed in the browser, typically React or Vue single-page applications or infinite-scroll lists. The initial HTML of such pages is often an empty shell, and directly scraping the HTML text yields no useful data; a headless browser (such as Playwright or Puppeteer) is needed to actually execute the scripts before reading the DOM. Login-state pages, on the other hand, require a valid session Cookie or Token and may also involve CAPTCHAs and risk controls. Both types of scenarios increase the engineering complexity and cost of scraping, and they are also the boundary where general-purpose web extraction tools tend to have fluctuating results.

Significance for AI Agent data pipelines

This tool reflects a current trend in Agent infrastructure: separating webpage understanding from the Agent's reasoning loop and handing it to a dedicated service that returns clean, verifiable structured results. The evidence-reference mechanism responds to another pain point—when an Agent makes decisions based on webpage data, users need to know where the data comes from and whether it is reliable. For teams building webpage data collection pipelines, such tools are worth including in the comparison list for technology selection.

One-line Summary

Apify's new tool turns webpages into structured JSON with evidence references using natural-language instructions.

Paragraph Summary

The article introduces the Webpage to JSON for AI Agents tool on the Apify platform, which attempts to solve the problems that LLM Agents are not good at parsing messy HTML and that traditional custom scrapers easily break due to redesigns. Users only need to provide a list of URLs and a plain English instruction to obtain JSON data with type declarations. Its core design is the evidence chain: every extracted value comes with a reference to the original page text, and fields for which no basis can be found are placed in unverifiedFields, used to identify LLM hallucinations. The tool supports JSON Schema for precise output constraints, listMode for batch extraction from listing pages, and a requireEvidence quality-check gate enabled by default, and can be integrated into Agent clients such as Claude and Cursor via the Python client or the MCP protocol. Billing is $0.03 per page for successfully extracted pages. The article also points out its limitations on complex pages, pages requiring login, or dynamically rendered pages, noting it is better suited to rapid validation and low-frequency collection.