Building a Search and Fetch Layer for LLM Agents: Caching, Content Extraction, and Cost Control

The cost of AI agents calling search and fetch APIs is shifting from "a few lines of requests code" to a bill that needs serious accounting. According to discussions on Hacker News, DeepSeek's search and fetch API is priced at $0.75 per 1,000 successful requests—cheap per call, but an agent needs 5 to 10 searches to answer a single question, and each search fetches 3 to 4 pages. The numbers add up quickly over time. Meanwhile, controversy over Meta's scraping terms reminds developers that scraping isn't just a technical issue—it also carries compliance responsibility.

A developer who rebuilt an internal project summarized this fetch layer approach. The core idea: don't let the agent access the internet directly. Instead, add a gateway in between that consolidates caching, budgeting, compliance, and content extraction into one place.

Overall Architecture: Gateway Pattern Instead of Raw Calls

The most common mistake is giving the LLM a fetch_url(url) tool that lets it make HTTP requests directly. The consequences are concrete: the same page gets fetched four times in a single session, 200KB of raw HTML (including navigation, footers, and tracking scripts) gets stuffed into context, and at the end of the month nobody can explain where the money went.

The right approach is to insert a Fetch Gateway layer. After the agent initiates a tool call, the gateway checks in sequence whether the cache hits, whether the daily budget allows it, and whether robots.txt permits it. Only when all checks pass does it actually make the HTTP request. After fetching the page, it extracts the main content, writes it to the SQLite cache, and returns the result to the agent. The agent always receives clean text with a length limit; when denied, it receives a structured reason rather than an exception that breaks the flow.

The key to this design is the boundary of responsibility: the agent only decides "what to fetch," while the gateway decides "whether it can be fetched, how much, and at what cost."

Responsible Scraping: robots.txt, User-Agent, and Content Extraction

The compliance part doesn't actually need extra dependencies—Python's standard library urllib.robotparser is sufficient. The author uses lru_cache to cache the parsed robots.txt result for each origin, avoiding repeated fetching. The policy for when robots.txt can't be read (allow or deny) is left to the business to decide.

For content extraction, trafilatura 2.0 was chosen. The author compared readability-lxml and newspaper4k and found trafilatura to be the most stable on news sites and Vietnamese blogs. Output format should be markdown rather than plain text—LLMs clearly understand heading levels and tables better, with almost no difference in token consumption.

The measured data speaks for itself: a VnExpress page's raw HTML is about 180KB, but after trafilatura extraction it's only about 6KB. If you're paying per token, this is the biggest saving in the entire pipeline—even exceeding the cost of the fetch API itself.

There are a few easily overlooked details in the code:

• The User-Agent must be identifiable and include contact information. Pretending to be Chrome might bypass some WAFs, but this is exactly the behavior that rules like Meta's terms target. Commercial products shouldn't do this.
• Hard truncate the output, for example limiting to 8,000 characters (about 2 to 3k tokens), which is enough to answer most questions.
• SSRF protection is mandatory: prohibit fetching private IP ranges (10.x, 192.168.x) and the cloud metadata endpoint 169.254.169.254. Once an agent falls victim to prompt injection, it could be tricked into reading the cloud host's metadata interface.

robots.txt is a plain-text protocol file in a website's root directory that uses directives like User-agent, Allow, and Disallow to declare which paths crawlers may access. It's not a legal statute, but it is the industry-recognized baseline for scraping etiquette. Commercial crawlers that ignore it risk bans or compliance disputes. Python's urllib.robotparser.RobotFileParser can parse this file per origin and provides can_fetch(ua, url) checks. Combined with lru_cache to cache parsing results, it avoids re-fetching on every request. Note that robots.txt itself may return 403 or time out. Whether to allow or deny in that case should be an explicit business policy written into configuration, not an implicit default.

SSRF (Server-Side Request Forgery) refers to an attacker inducing the server to make requests to internal addresses. In agent scenarios, the risk is amplified: prompt injection instructions embedded in page content or user input could cause the LLM to construct tool calls like http://169.254.169.254/latest/meta-data/, thereby reading the cloud host's temporary credentials. The core defense is to perform address resolution and validation before the gateway makes a request: reject private network ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8) and link-local addresses, and re-validate the actual IP after DNS resolution to prevent bypass via domain rebinding.

Caching and Budgeting: The Real Cost Savings

At $0.75 per thousand requests, a single request is indeed cheap, but an agent's looped calls can quickly push monthly volume to 50,000 requests—not to mention infinite retries caused by bugs. The author chose SQLite rather than deploying an additional service. With WAL mode enabled, it can easily handle hundreds of requests per minute.

The cache table uses the SHA256 of the URL as its key, with a TTL (1 to 6 hours for news, 24 hours or more for technical documentation). The budget table accumulates counts per day and only counts when an external request is actually made; cache hits are not counted. Under the team's internal Q&A workload, the cache hit rate is stable at 35% to 45%—users repeatedly asking about the same topic on the same day is normal. In other words, about 30 lines of code eliminate nearly half the fetch cost.

Another engineering habit worth adopting: when the budget is exhausted, return structured data like budget exceeded rather than throwing an exception. Newer LLMs can understand "I can't continue querying, answering based on existing information..." and wrap up gracefully instead of crashing the entire loop.

SQLite's WAL (Write-Ahead Logging) mode appends writes to a separate log file first, allowing reads to proceed concurrently and avoiding the read-write blocking of the default rollback journal mode. Enabling it only requires executing PRAGMA journal_mode=WAL;, and combining it with PRAGMA synchronous=NORMAL; significantly improves concurrent throughput while maintaining crash safety. For a fetch gateway handling hundreds of requests per minute, a single-file database with WAL is sufficient, saving the operational overhead of deploying Redis or Postgres. However, note that the WAL file grows with writes and requires periodic checkpoints, and it is not suitable for deployments where multiple machines share the same file.

Integrating with the Agent: Tool Descriptions Are Also a Cost Control Lever

Writing constraints clearly in the tool description works better than you'd expect. The author notes in the description: "Returns cleaned markdown, max 8000 chars. May return cached content up to 6h old. Do not fetch the same URL twice." This single line significantly reduced duplicate fetches.

It's also recommended to log each tool call to a separate table, including url, latency, cache hit, and token count, for periodic review of where the agent's money is going. A single sqlite3 fetch_cache.db "SELECT day, n FROM usage ORDER BY day DESC LIMIT 7;" shows the call trend for the last seven days.

Key Practices

If you only remember five things: don't let the agent make HTTP requests directly, even if the gateway is only 100 lines of code; use tools like trafilatura to extract content and hard-truncate by character count; cache with reasonable TTLs—SQLite plus WAL is enough for most small teams; set daily budgets and return denial reasons as data; respect robots.txt, use an identifiable User-Agent, and block SSRF.

Even if you switch to a paid API like DeepSeek^Ad, this gateway is still worth keeping—just swap fetch_clean for the vendor's client. Caching and budget control must remain in your own hands. The fetch layer is becoming agent infrastructure with both real costs and legal responsibilities, and it deserves the same seriousness as a database connection pool.

One-line Summary

Add a fetch gateway layer for LLM agents to unify caching, budgeting, compliance, and content extraction for cost control.

Paragraph Summary

Starting from DeepSeek's search and fetch API pricing of $0.75 per thousand requests, the article points out that a single agent Q&A often triggers dozens of fetches, with both costs and compliance responsibilities accumulating. The author advocates replacing direct HTTP requests from the agent with a Fetch Gateway: the gateway checks the cache, daily budget, and robots.txt in sequence, then fetches and extracts content with trafilatura, writing to a SQLite cache. Measured results show 180KB of raw HTML can be compressed to about 6KB—the biggest token saving in the pipeline. SQLite with WAL and TTL achieves a 35%–45% cache hit rate for internal Q&A, with about 30 lines of code cutting costs nearly in half. The article also emphasizes an identifiable User-Agent, SSRF protection, hard output truncation, and returning structured errors when the budget is exceeded so the LLM can degrade gracefully.