Models Update Every Week — How Do You Know Which One Fits Your Product? Notion AI's Head Shares the Case for Building Your Own Evals

Why "Building Your Own Evals" Has Become a Must for AI Product Teams

New models and agents are launching almost every week, at a pace that leaves product teams scrambling to keep up. GPT, Claude, and Gemini iterate in rapid succession, while open-source models continue to multiply — yet an uncomfortable reality persists: a high score on a leaderboard doesn't mean a model will actually perform well in your specific product context.

Sarah Sachs, Head of AI at Notion, will be presenting at the upcoming Forge event in San Francisco, where she makes a compelling case: rather than chasing the latest model releases, product teams should build their own dedicated evaluation systems (evals). This deceptively simple recommendation cuts right to the heart of a pain point that countless AI teams are feeling right now.

The Limits of Public Leaderboards: High Scores Don't Equal Real-World Performance

Model benchmarks — like MMLU and various public leaderboards — measure general capabilities, but products deal with specific tasks. A model that excels at mathematical reasoning may struggle with your customer service conversation summaries. An agent that leads on a comprehensive leaderboard might make frequent mistakes inside your actual workflow.

The core issue is this: there's a fundamental gap between general benchmarks and your real business scenarios. When a new model claims to "achieve SOTA" every single week, without a measuring stick calibrated to your own needs, teams easily fall into the trap of swapping model after model without being able to say clearly whether anything actually improved.

This is the foundational reason Sachs advocates for building your own evals — only you truly understand what your users need and what your product's failure cases look like.

MMLU (Massive Multitask Language Understanding) is one of the most widely cited LLM benchmarks, covering multiple-choice questions across 57 subjects including math, history, law, and medicine, used to assess a model's breadth of knowledge. "Leaderboard" refers broadly to public rankings such as the Hugging Face Open LLM Leaderboard and LMSYS Chatbot Arena. These benchmarks share a common trait: they prioritize breadth and generality, with fixed, publicly available test sets — which also means model vendors have incentive to optimize specifically for these questions, a phenomenon known as "benchmark gaming." More fundamentally, these benchmarks cannot capture the fine-grained requirements of vertical use cases: things like legal document formatting conventions, domain-specific terminology accuracy, or semantic understanding of how a particular type of user phrases a question. SOTA (State of the Art) is a common AI abbreviation referring to the best-performing model on a given benchmark at any given time — but there is often a significant gap between "achieving SOTA" and "performing better in your product."

What Building Your Own Evals Actually Solves

An evaluation system purpose-built for your product delivers certainty in at least three key areas:

1. Turn Model Selection into a Data-Driven Decision

When a new model launches, your team can run it directly against your own eval set and use real task-performance data to decide whether to switch — rather than relying on social media buzz or someone else's subjective experience. This transforms "should we upgrade models?" from a gut-feel call into a quantifiable judgment.

2. Continuously Monitor Production Quality

Evals aren't just a one-time tool for model selection. As prompts are adjusted, model versions are updated, and user input distributions shift, product quality can drift quietly in the background. A continuously running evaluation catches regressions early, preventing the scenario where "we fixed one thing and broke another."

3. Align the Team on What "Good" Means

At its core, an eval set makes explicit and standardizes what a passing output looks like. It gives product, engineering, and design teams a shared language for quality, cutting down on unproductive subjective debates.

In engineering practice, an eval set typically consists of three components: input samples (representing real user requests), expected outputs (reference answers defined by human annotation or rules), and evaluation criteria (which can be exact matching, similarity thresholds, or an LLM-as-judge approach where another LLM acts as the evaluator). Regression detection is one of the most core values of continuous evaluation — whenever a prompt, model version, or system parameter changes, automatically running the eval set lets you quantify "which cases improved and which degraded with this change," preventing unintentional breakage of stable capabilities while optimizing another dimension. Building an eval system carries a high upfront cost, but once established, it becomes a force multiplier for team iteration speed, enabling engineers to experiment more boldly while retaining the confidence to roll back quickly.

What This Means for AI Product Teams

Sachs's insights come from first-hand practice at Notion — as a productivity tool deeply integrated with AI capabilities, Notion is acutely sensitive to model quality. Her perspective distills into a clear action path:

• Start by defining your core tasks, and collect input/output samples from real user scenarios;
• Build your eval set and set explicit metrics, whether that's accuracy, relevance, or human scoring;
• Integrate evals into your iteration workflow, so every model or prompt change has to "pass the test."

In an era where the model layer is evolving rapidly and differentiation is increasingly hard to find, a product's moat increasingly depends on how well you use models, not which model you use. And the ability to evaluate is the critical link that converts model capability into product value.

Closing Thoughts

Sarah Sachs will elaborate on Notion's evaluation practices at the Forge event in San Francisco on November 3rd. For every team being swept along by the relentless pace of weekly model releases, this may be a timely reminder worth pausing to consider: what truly matters isn't keeping up with every new release — it's building the capability to judge what actually works for your product.