A Third of Common Crawl Index Lookups Get No Answer: Lessons from Five Months of Monitoring

A developer running a small script on Cloudflare Workers — sending one routine query per day to the Common Crawl index to check whether a handful of their site's URLs had been captured in the latest crawl snapshot — expected a cheap, boring health check. Instead, five months of data revealed something far more interesting than coverage rates: the reliability of the index service itself.

An Unexpected Finding from a Boring Script

The workflow was straightforward: fetch the collections list once, run a CDX query for each URL, write the results to a table. The author's original concern was inclusion rates. What turned out to be more interesting was the failures.

Starting from May 15, across 173 snapshots and 1,889 total queries, 605 returned no answer at all — 32% of all queries. The breakdown by failure type:

• 344 × HTTP 503
• 103 × HTTP 504
• 80 × HTTP 521
• 52 × timed out by the author's own timeout logic
• 18 × 502, 7 × 500, 1 × 520

Even more striking: out of 173 snapshots, 11 had every single query fail. October was the worst month so far, with 79 failures out of 132 queries.

Common Crawl is a nonprofit organization that has been crawling the web at scale since 2008, producing petabytes of archived web data each month — freely available to researchers and developers. The CDX API (Capture Index API) is the standard interface for querying its index metadata: submit a URL to check whether it was captured, when, and with what HTTP status — though it doesn't return the page content itself. Common Crawl operates on limited server resources as a public-interest project, making it prone to overload during global traffic peaks. HTTP 521 is a Cloudflare-specific error code indicating that the origin server refused the connection — typically meaning the origin is down or unresponsive, a lower-level failure signal than 503/504.

Why 04:00 UTC Is the Worst Window

The cron job was set to fire at 04:00 UTC. Breaking down the data by time of day revealed a clear pattern:

• Between 03:00–06:00 UTC: 491 failures out of 1,339 queries — roughly 37% failure rate
• All other hours: 114 failures out of 550 queries — roughly 21%

The author was appropriately cautious about this comparison, acknowledging a selection bias: most queries outside the scheduled hour were manual reruns, typically triggered after seeing the morning failures — meaning they ran later in the day, when the upstream service may have already recovered. That self-awareness is commendable. Still, the author notes this directional pattern has held consistently across five months.

A Three-Morning Postmortem from October

The real design flaws were exposed by three consecutive days in October.

On the 10th and 11th, the script never even got to the URL query stage. The collections list request returned 521 immediately, and that call had no retry logic — so the entire phase failed and nothing was written. By contrast, URL queries had been given retry logic back in August; the collections list call had been overlooked.

The author then added three attempts to the collections list call, with 10-second and 20-second delays. On the 12th, that step finally succeeded — and then all nine URL queries that followed returned 521 anyway.

After each of the first two morning failures, the author reran the script in the afternoon: all 12 queries succeeded on the 10th, and 10 of 12 succeeded on the 11th. Same code, same URLs, just a few hours later — completely different results. At the time of writing (around 06:10 UTC), a plain curl returned a 200 for the collections list, but one URL query was still returning 504.

The Key Lesson: Short Retries Can't Fix Long Outages

This experience yields two practical takeaways.

Retry intervals must match the timescale of the failure

Retries spaced seconds apart can recover from occasional dropped requests. Against an index that's been down for hours at a stretch, they're useless. When upstream enters a bad window, the right fix is "try again in a few hours," not "try a third time in 20 seconds." This is the next change the author has planned for the pipeline.

This principle maps directly to the concept of backoff strategy in reliability engineering. Exponential backoff handles transient network jitter well. For broader service unavailability — brownouts or blackouts — the more appropriate pattern is a circuit breaker: once consecutive failures exceed a threshold, stop attempting and wait through a longer cooldown period, rather than repeatedly hitting a wall at fixed or exponential intervals. The author's planned "try again in a few hours" is essentially manual circuit breaking — extending the retry window from seconds to hours to match the actual recovery timescale of the upstream service.

Failures must be recorded as errors, not as "not included"

Another design decision the author is glad was made early: a failed query is stored as error, not as "not included". Coverage rates are only calculated over URLs that actually returned an answer; when the response count is too low, the metric is left blank rather than showing a potentially misleading number.

This distinction mattered enormously in October. The accurate reading for the 11th is "7 out of 10 URLs that answered were included" — not "7 out of 12 URLs were included." Only the first is true. If errors were silently collapsed into "not included," October's data would look like the site was being dropped from Common Crawl's index — when in reality, the index simply wasn't picking up the phone.

What This Means for Anyone Polling External Data

What reads like a mundane ops log actually distills a widely overlooked principle: before asking what queries returned, ask how many queries returned anything at all.

For any system that polls an external data source on a schedule — whether a crawl index, a third-party API, or a public dataset — conflating "no response" with "negative answer" will corrupt every downstream analysis. A 32% failure rate, silently swallowed, produces metrics that are fundamentally misleading. The author has made all collected snapshots publicly viewable on their site's research dashboard — the kind of reproducible, auditable transparency that data engineering should always aim for.