Vercel's Chief of Software Looks Back: The Evolution of Agent Building — From Multi-Agent Chains to File System Agents

At the AI Engineer conference, Vercel's Chief of Software, Andrew, shared a year-long "agent explosion" experiment he led internally at Vercel. Starting from a massive prompt that copy-pasted SQL, all the way to the open-source agent framework EVE, this journey hit nearly every pitfall in agent development today — and revealed an increasingly clear path toward best practices.

From "A Computer on Every Desk" to "An Agent on Every Desk"

In 1980, Bill Gates envisioned "a computer on every desk and in every home." About a year ago, Andrew and Vercel's CTO posed an analogous question: could you have an agent on every desk?

At the time (back in the era of Sonnet 4, when model capabilities were far weaker than today), this was a remarkably forward-thinking idea. Today, agents are primarily used for coding and technical workloads, but vertical applications in areas like design and product management are beginning to emerge. To validate this direction, Andrew spoke with teams across Vercel — marketing, sales, finance, legal — and asked them, "What part of your job do you hate the most?" The most compelling use case came from the data team: a lean team under enormous growth pressure, where Vercel's pace of growth far outstripped their ability to handle data. Every time marketing or sales had a question about customers or products, data scientists had to drop everything, write queries, run analyses, and deliver recommendations — seriously dragging on overall productivity.

Three Architectural Iterations: Giant Prompts, Multi-Agent Chains, and Monolithic Memory Management

The first version was remarkably simple: export Snowflake's schema, paste it into a system prompt, add the question, let the model generate SQL, and then Andrew would manually copy-paste it to execute. This rough experiment built initial confidence — today's models aren't perfect, but given reasonably structured context, they can produce usable SQL. The key was doing the surrounding "context engineering" well and giving the model more guardrails.

The second version, codenamed D0, broke down a data scientist's workflow into distinct phases: handle the question, explore the semantic layer and join patterns, execute SQL, retry on failure or excessive cost, then report and visualize. They designed a chain of specialized agents accordingly — a query agent, planning agent, execution agent, and so on — each with its own system prompt and precisely scoped tools (for example, the planning agent only had two tools: read entity YAML and search schemas).

This version achieved an end-to-end loop from question to answer, but the team quickly hit a wall. They realized the fatal flaw of multi-agent chains: downstream agents only receive summaries and small snippets from upstream agents, causing massive information loss in transit. So the third version shifted to a "single giant agent" — one AI call with max steps set to 100, letting it manage its own internal state, switching between planning, building, executing, and reporting, while being able to look back at its own previous actions and self-correct. When execution or joins failed, it could step back, read more, and explore further.

Chain-of-Agents architecture is one of the dominant paradigms in agent system design today: decomposing complex tasks into multiple specialized sub-agents, each handling a specific stage of the pipeline, with "context summary passing" connecting upstream and downstream agents. The advantages are clear separation of responsibilities and easier debugging — the tool set at each node can be tightly constrained, reducing the risk of the model "overstepping." But its core flaw is exactly what the article reveals: every handoff compresses context, with original information degrading layer by layer until little remains by the time it reaches the final agent. This closely mirrors how information distorts as it passes through organizational hierarchies. The Monolithic Agent sits at the opposite extreme: all reasoning steps are handled by the same model call, using a large token window to preserve the complete history so the model can reference any earlier state. The tradeoff is higher cost, longer single-call latency, and greater difficulty optimizing individual steps. The choice between these two architectures is fundamentally a tradeoff between "information completeness" and "system modularity" — and there is no universal optimal solution in the industry today.

Outclassed by Claude Code and Opus 4.5

The team was fairly confident in the monolithic version, with an eval pass rate of around 30%, so they sent it to a small group of trusted colleagues for a trial. The feedback was: "it was terrible" — real users asked questions far beyond the scenarios they had manually mapped, and continuing to enumerate scenarios by hand clearly wouldn't scale.

The turning point was the release of Claude Code and Opus 4.5. Andrew was blunt: compared to the agent they had hand-crafted, "Claude Code plus Opus 4.5 is basically AGI," capable of answering most questions almost effortlessly. When they did a post-mortem on "what exactly did we get wrong," they arrived at a core conclusion: the biggest unlock was the file system. Claude Code has a minimal tool set — list file, read file, run bash — but rather than providing highly prescriptive tools, it lets the model explore freely and exhibit emergent behavior. These are precisely the capabilities that agents have been thoroughly trained on.

File System Agents: Eval Scores Doubled Overnight

Building on this insight, the team rebuilt D0 with a "Claude Code-style" approach: letting the agent run in a sandbox, loading the entire semantic layer into the sandbox, letting the agent freely explore via bash, read file, and write file, then "sprinkling in" a few Vercel-specific tools. This was "the single biggest unlock in history" — eval scores doubled instantly. And the implementation was surprisingly simple: use a bash tool helper from npm to mount a sandbox, put the files in, and let it read, write, and execute.

Andrew then wrote a retrospective blog post that drove 70% of Vercel.com's traffic in the week it was published. The next optimization was "skills": the team discovered that thousands of daily queries followed highly repetitive patterns — aggregations, product lookups, billing information — with only a handful of query structures. So they set up a periodic task to distill recent queries into skills; they've now accumulated about 100 skills. Each new agent run no longer starts from scratch — it begins with accumulated contextual knowledge already in place. Vercel also released Skills SH, a tool for finding and running agent skills.

The "evals pass rate" here is a core engineering metric for measuring agent quality. Evals (evaluation sets) typically consist of a set of test cases with ground-truth answers or automatic scoring, covering typical inputs, edge cases, and known failure scenarios. Similar to unit tests in traditional software, evals let teams quantify regressions or improvements after each change. In agent development, because model outputs are inherently non-deterministic, eval design is especially important — covering both "correctness" (e.g., can the SQL execute? is the result accurate?) and "robustness" (e.g., does the model hallucinate on ambiguous questions?). The jump from ~30% to ~60% pass rate described in the article corresponds to a systemic architectural change rather than fine-tuning, demonstrating that the gains from giving the model "free exploration" via the file system are structural — not marginal optimizations.

EVE: The "Next.js" for Agents

Andrew pointed to a recurring pattern: at every stage of building D0, Vercel colleagues who were "agent-curious" would fork his code to build their own agents, and at every step there existed better approaches that weren't yet known. This prompted the team to ask — could people start from the final insights, rather than having to reinvent the wheel from simple prompts or first principles every time?

The answer is EVE, an open-source framework released two weeks ago, positioned as "the Next.js for agents." Just as Next.js uses file system conventions to automatically declare infrastructure, EVE lets developers simply create a skills folder, a tools folder, and a channels folder — and the framework knows how to assemble them into an agent.

EVE's agent model includes a runtime and channels: the runtime handles persistence, isolated execution environments, model calls, and connections. The framework is designed with open source at its core — it can plug in your own adapters for Postgres, OpenAI Responses API, Docker, and more — while also deploying conveniently to Vercel, with support for Vercel Workflows (persistence), Sandbox (secure execution), and the newly released Vercel Connect (which generates short-lived OIDC tokens for connections). The team rewrote the entire D0 using EVE, and the final file structure is remarkably clean — a set of system instructions, a few skills, a few tools.

OIDC (OpenID Connect) is an identity authentication protocol built on top of OAuth 2.0, widely used for short-lived token authorization between services. The "short-lived OIDC tokens" mentioned here are an important concept in cloud-native security practices: compared to long-lived API keys, short-lived tokens have a minimal blast radius if leaked, and can be scoped to specific workloads and time windows — dramatically reducing the security risk when agents access external services. Vercel Connect dynamically issues a token for each agent run and automatically invalidates it when the run completes, baking security into the framework layer so developers don't need to manually manage credential rotation. This is also one of the key dimensions that separates enterprise-grade agent frameworks from personal demos — observability, isolated execution environments, and least-privilege tokens together form the security baseline for production readiness.

The Real Moat for Vertical Agents: Company-Specific Knowledge

Before the official launch, EVE was already being tested with select beta customers. Partner Aura used EVE to rebuild a "mini Claude"-style agent from scratch for automatically visiting websites, installing and testing services — and achieved fewer steps, higher success rates, and better insights compared to off-the-shelf Claude Code. Deployed on Vercel, it comes with built-in observability: you can inspect every run, every tool call, every step, plus estimated costs and optimization suggestions.

Andrew emphasized a core point: off-the-shelf vertical agents (like ones purpose-built to run Snowflake queries) are great and worth trying, but what truly makes an agent excellent is a large amount of company-specific knowledge. Vercel, as a web company, has deep knowledge of the relationships between customer site properties and knows when to query what — things generic tools can't replicate. Vercel now has about 20 agents internally that have reached PMF, covering scenarios from marketing retrospectives and sales outreach to first-pass legal contract redlining and data queries. The data team has been freed up to spend time optimizing Snowflake performance and onboarding new data sources. Andrew recommends that companies of all sizes try building their own agents, loading in as much company-specific knowledge as possible — and EVE is one of the best starting points available today. Developers can get started with templates at eve.dev.