Open-Source seo-agent: A Cloudflare-Based AEO Edge Engine That Helps AI Crawlers Understand Your Website

As SEO Enters the AEO Era: AI Crawlers Are Changing Content Distribution Rules

The rules of search optimization are undergoing a fundamental shift. In the past, we focused on how to rank content higher on Google (SEO). Now, as AI applications like ChatGPT, Perplexity, and Claude become new information gateways, a new concept is emerging — AEO (Answer Engine Optimization), along with the related GEO (Generative Engine Optimization).

The rise of AEO stems from a fundamental change in how users access information. Traditional search engines (like Google) return a list of links, requiring users to click through and filter information themselves. Answer engines (like ChatGPT and Perplexity), on the other hand, synthesize a complete answer directly and selectively cite sources. This means content creators face a different competitive dimension — it's no longer about fighting for "ten spots on page one," but about earning the qualification to be "selected by AI as an answer source." GEO goes further, focusing on how to achieve higher citation weight in generative AI outputs, including factors like authority signals, structural quality, and factual density.

The core issue is this: AI crawlers scrape web pages in fundamentally different ways compared to traditional search engine crawlers. Traditional crawlers (like Googlebot) have evolved over two decades to develop robust JavaScript rendering capabilities, able to execute JS code and index dynamically generated content. However, AI crawlers (such as OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl) have significantly different crawling strategies — their goal isn't to build a web index, but to extract high-quality training data or real-time retrieval corpora. As a result, AI crawlers prefer plain text or Markdown format content that can be directly parsed, while they often struggle with single-page applications (SPAs) that rely heavily on JavaScript rendering, frequently capturing nothing more than an empty HTML skeleton. This means that even if your website performs excellently in traditional search, it may be completely absent from AI engines' citation lists.

Recently, a developer open-sourced a project called seo-agent on Reddit (GitHub: citeworthyio/seo-agent), attempting to systematically solve this challenge at the edge server level. This Cloudflare-based automated AEO/SEO proxy provides a complete AI-era optimization solution for large-scale content sites.

seo-agent Technical Architecture

seo-agent is positioned as an edge server based on Cloudflare. Cloudflare's edge computing platform (primarily implemented through Cloudflare Workers) is a serverless runtime environment with code deployed across more than 300 data centers worldwide. Unlike traditional centralized servers, edge servers are physically closer to the request source of end users or crawlers, enabling them to intercept and process requests before they reach the origin server. seo-agent leverages this architectural characteristic to serve as a middleware layer between user/crawler requests and the origin, performing intelligent routing and content adaptation without modifying any origin code.

Three Core Capabilities

According to the project documentation, seo-agent addresses three main problems:

• Providing Markdown content for AI Agents: When AI crawlers are detected, it dynamically serves clean Markdown-formatted content instead of raw HTML, thereby increasing the probability of being cited by AI. Markdown, as a lightweight markup language, features clear structure, explicit semantics, and low parsing overhead — making it ideal for direct consumption and comprehension by AI models.

• Rendering SPAs and JS applications for AI crawlers: Single-page applications (SPAs) are the mainstream architecture in modern frontend development. Applications built with React, Vue, Angular, and similar frameworks typically render page content dynamically on the client side via JavaScript. When a browser loads an SPA, the initial HTML usually contains only an empty <div> container and several JS file references — actual content is generated only after JS execution. For AI crawlers that don't execute JavaScript, they "see" an essentially blank page. seo-agent performs pre-rendering at the edge layer, providing crawlers with fully rendered content snapshots. This avoids invasive modifications to existing frontend architectures and eliminates the expensive cost of migrating SPAs to server-side rendering (SSR) or static site generation (SSG).

• Assessing AEO/SEO posture: By connecting to your Google Search Console in read-only mode, it helps you understand your site's search and AI performance across all pages, while tracking traffic from various AI crawlers. This includes identifying access frequency and crawling behavior of GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers, providing data-driven insights for optimization strategies.

Fail-Open Mechanism Ensures Site Stability

A commendable engineering design is its "fail open" mechanism — if anything goes wrong at the edge layer, requests fall back directly to the origin server. "Fail open" and "fail closed" are two core safety strategies in distributed system design. Fail closed means that when middleware encounters a fault, all requests are rejected to ensure security (common in firewalls and authentication systems). Fail open means that during failures, requests are allowed to pass through, prioritizing service availability. In CDN and edge proxy scenarios, fail open is the more sensible choice — because the edge layer provides enhancement features rather than a security barrier. If the enhancement layer fails, having requests reach the origin normally is far better than making the entire website inaccessible. This design embodies the engineering philosophy of "progressive enhancement": edge optimization is a nice-to-have, not a prerequisite for system operation. For large-scale production sites, this is a critically important safety guarantee.

Broad Platform Compatibility

Another major advantage of seo-agent is its host-agnostic nature. The project claims it works with any host that can serve as an origin, including Vercel, Shopify, Cloudflare, DigitalOcean, and other mainstream platforms. This design lowers the barrier to adoption, allowing sites with different technology stacks to benefit. The underlying principle leverages the architectural characteristics of a reverse proxy — the edge layer only cares whether the origin can respond to HTTP requests, not about the specific implementation technology behind it.

Dual-Track Model: Open-Source Engine and Commercial Product

seo-agent employs an open-source commercialization strategy worth studying. The author explicitly states that the open-source engine you download today is the exact same engine powering their commercial product citeworthy.io — the open-source and commercial versions remain in lock-step, with no feature cuts or delayed releases.

This strategy of complete synchronization between open-source and commercial products represents a fairly aggressive transparency commitment in the open-source commercialization space. More common models include "open-source core + closed-source enterprise edition" (like GitLab and early Elasticsearch) or delayed open-source releases. Complete synchronization means the commercial product's differentiation relies entirely on managed hosting, technical support, and user experience at the service layer, not feature differences. The benefit of this model is building community trust and attracting developer contributions — similar successful examples include Plausible Analytics and Cal.com.

Two Usage Paths

Self-hosted approach: For teams with technical capabilities, you can obtain the open-source engine directly from GitHub and deploy it yourself. The repository is fully open, and PRs and issue reports are welcome.

Managed service approach: For users who want to skip configuration hassles, citeworthy.io offers a freemium product:

1. Verify domain ownership through a simple TXT record to preview optimization effects without modifying your existing site. TXT record verification is a standard domain ownership proof method, similar to the verification mechanisms used by Google Search Console and SSL certificate authorities, ensuring only the actual domain controller can enable the proxy service.
2. If you decide to proceed, a single DNS change connects your site to the edge network. Essentially, this involves modifying your domain's CNAME or A record to point traffic at seo-agent's edge nodes instead of directly at your origin server. All changes can be previewed, approved, and tracked. The entire process requires zero modifications to your origin's code or configuration.

This progressive "see the results first, then decide whether to connect" experience significantly reduces decision risk for users. The author also states they're willing to provide free Pro licenses to users who genuinely use the product and provide effective feedback.

Real-World Results and Scale Validation Data

For open-source projects, real data is often the most convincing evidence. According to the author, the commercial engine currently serves 7 sites under their portfolio, several of which are large-scale with significant traffic. Over the past 14 days, these sites achieved approximately 13% traffic growth.

Even more indicative of system stability is its operational scale — the platform currently processes over 1 million (1MM) edge requests per day and already hosts some third-party sites that "discovered it in the wild and connected on their own." For a freshly open-sourced project, this kind of production-grade operational data provides considerable trust backing.

It's worth noting rationally that the 13% traffic growth comes from short-term observation of the author's own sites with limited sample size — specific results still need validation from more independent users. However, from an engineering completeness and operational scale perspective, this project has far exceeded "toy-level" demo territory.

Why AEO Deserves Attention from Content Site Operators

The emergence of seo-agent reflects a trend that's taking shape: The target audience for content distribution is expanding from humans and search engines to include AI Agents.

As more users access information through AI assistants rather than clicking search results directly, whether your content can be "understood" and "cited" by AI will directly determine brand and traffic visibility. According to multiple industry studies, tools like ChatGPT and Perplexity have reached hundreds of millions of monthly active users, and information consumption on these platforms is gradually displacing traditional search behavior. Traditional SEO focuses on rankings and clicks, while AEO focuses on the qualification and frequency of being cited by AI — this represents a fundamental leap from "being seen" to "being trusted as a source."

For content sites, e-commerce platforms, and media organizations, investing in AEO early may be a necessary move to address the paradigm shift in search. Tools like seo-agent that operate at the edge layer while balancing open-source transparency with commercial viability offer a pragmatic path for this transition.

Conclusion: A Starting Point for Content Optimization in the AI Search Era

seo-agent is an open-source initiative for content optimization in the AI era. Built on Cloudflare's edge server infrastructure, it solves a series of emerging pain points including AI crawler Markdown delivery, SPA rendering, and AEO posture assessment. Its fully synchronized open-source engine and commercial product, robust fail-open design, and million-request-per-day operational scale all make it a project worth attention from content site operators.

If you're thinking about how your website should position itself in the AI search era, this project offers at least a hands-on starting point for experimentation. Project URL: github.com/citeworthyio/seo-agent.