Vercel's Agent Evolution: From Failure to Double the Score with a File System Approach

Vercel's Chief Software Officer Andrew recently shared a complete retrospective of how the company's internal data science agent D0 went from an initial failure to doubling its evals score. This hands-on journey not only reveals the critical turning points in building production-grade AI agents, but also directly inspired EVE — an open-source agent framework Vercel released two weeks ago. For any team looking to deploy enterprise-grade agents in the real world, this hard-won experience is an invaluable roadmap.

From Mega Prompt to Multi-Agent: The Failed Experiment

Vercel is rooted in web infrastructure, helping developers deploy websites and applications without worrying about the underlying plumbing. But Andrew noticed demand was shifting — people were moving from building pages to building agents. About a year ago, in the era of Sonnet 4, he and the CTO had an idea: could they achieve "an agent on every desk" the way computing once put "a computer on every desk"?

He made the rounds across Vercel's marketing, sales, finance, and legal teams, asking everyone the same question: "What part of your job do you hate the most?" The most compelling pain point came from the data team. It was a lean team, but Vercel was growing fast, with a constant flood of data from customers, analytics, and sales. Whenever marketing or sales had a question about a customer or product, data scientists had to drop everything — write queries, run processing, do analysis, make recommendations. It was a productivity killer.

The first prototype was dead simple: Andrew grabbed a Snowflake schema export, pasted it into a system prompt, added the question, had the model generate SQL, then manually copy-pasted it to execute. This rough experiment gave him a sliver of confidence — models weren't quite good enough yet, but with better context engineering and guardrails, it might be viable.

Hitting the Multi-Agent Wall

The second version, D0, decomposed a data scientist's real workflow into multiple specialized agents: a query agent, a planning agent, an execution agent, and a reporting agent, chained together in a pipeline. Each agent had its own system prompt and a tightly scoped set of tools — for example, the planning agent only had access to two tools: "read entity YAML" and "search schema."

This architecture was genuinely better than manual copy-pasting — it achieved an end-to-end loop from question to answer. But they quickly hit a wall. The problem: each downstream agent only received a summary and small snippet from upstream. It couldn't look back at the full execution history or self-correct.

This led to a key conclusion: what they actually needed was a single large agent with full context that could manage its own memory. This mega-agent would switch states internally — planning at one moment, building at another, executing at another, reporting at another. By using one large call with a max of 100 steps, it could autonomously manage its own state based on current progress. The biggest advantage: when an execution or join failed, it could backtrack, re-explore, re-read, and figure out what went wrong.

Multi-agent Pipeline was an early common pattern for enterprise LLM deployments: break a complex task into subtasks, assign each to a dedicated agent, and chain them together via message passing. The theoretical advantages are clear separation of responsibilities, shorter and more focused prompts, and limited blast radius per step. But the fatal flaw is information loss: each handoff between agents compresses the upstream output into a summary, discarding the full intermediate reasoning chain, execution logs, and contextual details. Downstream agents work from an incomplete picture — unable to review full history for self-correction or do genuine long-horizon planning across steps. The difference from human collaboration is that human collaborators can always ask upstream colleagues for clarification or pull original materials; a fixed-topology agent pipeline cannot. This is exactly the wall Vercel hit, and the core motivation for pivoting to a single large agent.

A Reality Check from Real Users

The team was fairly confident. They handed this "pretty capable" system to a handful of trusted internal colleagues — they were worried about it landing in the wrong hands or running on critical workloads. The feedback was blunt: it was terrible.

They thought scoring 30% on evals was solid, but were completely unprepared for the variety of questions real users would ask. And manually mapping out all those edge cases wasn't a scalable approach. It was a brutal but real lesson: there's often a vast gulf between high internal benchmark scores and real-world performance.

The File System Agent: The Real Unlock

The breakthrough came with the launch of Claude Code and Opus 4.5. Andrew described it as "basically AGI compared to the agent we'd hand-rolled" — it could answer nearly all of their questions almost effortlessly.

As the team reflected on what they'd done wrong and why Claude Code was so powerful, they realized the biggest unlock was surprisingly simple: it's just a file system. A minimal toolset — list file, read file, run bash — plus a few data-specific tools. The key was that these were tools the agent had been thoroughly trained on, allowing the model to freely explore and write work products where needed. Claude Code didn't prescribe a rigid toolset; it let emergent behavior flourish.

So they rebuilt D0 in the spirit of Claude Code: running in a sandbox, dumping the entire semantic layer into it, letting the agent freely explore via bash, read file, and write file, then "sprinkling" a few Vercel-specific tools on top. The improvement was unprecedented — from a single agent to the Claude Code SDK, to a file system agent tailored for their use case, evals scores doubled.

The implementation was surprisingly simple: give it a bash tool (there's a helper on NPM called bash tool), attach it to a sandbox, mount a file system for reading, writing, and executing — done. The blog post Andrew wrote about this contributed 70% of Vercel.com's traffic in the week it was published.

Claude Code is Anthropic's terminal-based coding agent, and its core design philosophy is limiting the agent's action space to the primitives developers know best: file read/write and shell execution. There's important training logic behind this minimal toolset — during pretraining and RLHF, the model was exposed to massive amounts of real code and operational records using these tools, giving it exceptional generalization ability over commands like ls, cat, and bash. By contrast, custom tools hand-designed for specific business scenarios (like "query schema" or "generate report") often lack sufficient training data, and the model's understanding of their edge cases is far less robust than its grasp of standard file system operations. A sandbox is a security mechanism that isolates the execution environment, letting the agent run arbitrary code without affecting the host system. Vercel's approach of dumping the business semantic layer into the sandbox file system — letting the agent explore data context the way it would explore a codebase — is fundamentally borrowing capability pathways the model has already been thoroughly reinforced on, rather than forcing it to learn a new tool language.

Skill Distillation: Never Starting from Zero

Once D0 was opened up to all of Vercel, thousands of queries poured in daily — customer metrics, sales metrics, NPM download counts, and more. The team noticed these queries were structurally very similar: only a handful of aggregation patterns, plus recurring lookups for product and billing information.

So they built a scheduled job that distills recent queries into "skills." They've now accumulated around 100 skills, covering everything from aggregate analysis to querying specific people's data. The mechanism works well: each new agent run normally starts from scratch with almost no preset context beyond the semantic layer and system prompt. With skills, it begins with a wealth of already-distilled contextual knowledge. Vercel also launched a Skills SH tool, which has become a popular way to find and run agent skills.

This "skill distillation" mechanism corresponds in AI system design to a hybrid of Retrieval-Augmented Generation (RAG) and few-shot examples. Traditional RAG vectorizes an external knowledge base and retrieves relevant chunks on demand to work around context window limits. Vercel's skill system goes further — it doesn't just store knowledge fragments, it preserves validated complete execution paths (including SQL queries, analytical approaches, and report formats). When relevant skills are injected into context at runtime, it's like handing the agent a seasoned employee who has "done this exact type of task 100 times before." The value of this mechanism is that it makes the system's tacit knowledge explicit and continuously accumulating, creating a compounding effect: the more it's used, the richer the skill library, and the higher the starting point — and success rate — for each subsequent run. This is also one of the core moats of building a vertical agent in-house versus relying on a generic product.

EVE: The Next.js for Agents

This journey — from a simple prompt all the way up to a production system — made Andrew realize something. Along the way, colleagues who were "agent curious" kept forking his D0 to build their own agents, but each time they had to reinvent best practices from scratch or first principles.

This gave birth to EVE — an agent framework released two weeks ago, positioned as "the Next.js for agents." Just as Next.js uses file system conventions to define infrastructure, EVE lets you simply create a skills folder, a tools folder, and a channels folder, and the framework knows how to assemble an agent. It's designed to be open source, with pluggable adapters for Postgres, the OpenAI Responses API, Docker, and more, while also supporting one-click deployment to Vercel, with access to Vercel Workflows (persistence), Sandbox (safe execution), and the newly released Vercel Connect (short-lived tokens).

The team rewrote all of D0 using EVE, and the file system structure became remarkably clean: a set of system instructions, a few skills, and a few tools combine to form a functional, easily iterable agent. Deploying to Vercel also provides built-in observability — every agent run, tool call, step of execution, and estimated cost is visible at a glance.

Partner Aura used EVE to rebuild a "mini Claude"-style agent from scratch — one that automatically visits websites, installs and tries out other people's services. Compared to using Claude Code directly, it required fewer steps, achieved higher success rates, and produced stronger insights.

Next.js is a React full-stack framework built by Vercel, and its core innovation is replacing tedious manual configuration with file-system-based routing conventions: put a file in the pages/ directory and the framework automatically generates a route; put it in api/ and it becomes an API endpoint automatically. This "convention over configuration" philosophy dramatically reduces the cognitive overhead of web development, letting developers focus on business logic rather than infrastructure assembly. EVE brings this same philosophy to the agent space: the presence of skills/, tools/, and channels/ folders is the configuration — the framework understands what they mean and handles the assembly. This stands in contrast to mainstream agent frameworks like LangChain and LlamaIndex, which rely on extensive Python code to explicitly declare component relationships. The latter is flexible but steep; the former is more constrained but extremely fast to pick up, especially for web developer teams who already have Next.js experience.

Conclusion: The Value of Building Vertical Agents In-House

Andrew's core argument is this: off-the-shelf vertical agents (like products specifically built to run Snowflake queries) are decent and worth trying, but to really "squeeze the juice," you need to build your own and inject as much company-specific knowledge as possible. As a web company, Vercel has a deeper understanding of its customers' site properties — it knows when to query what, and what connects to what. That domain knowledge is something generic products simply can't provide.

Today, Vercel has around 20 internal agents that have reached strong PMF, spanning marketing retrospectives, outreach target filtering, first-pass legal contract redlining, and data science queries. The data team has been freed up to optimize Snowflake performance and onboard new data sources, with productivity at an all-time high. For companies of any size, the barrier to using agents to automate tasks you don't want to do or that take too long — HR, finance, sales, and beyond — has never been lower.