Metaview in Practice: Building a Prompt System That Evolves With User Preferences

In a technical talk hosted by Anthropic, Nick Mayhew, a product engineer at AI recruiting software company Metaview, walked through how they built a "self-evolving" prompt system for automated resume screening. This case study is more than an engineering retrospective — it reveals a core problem that most LLM applications overlook: when user judgment sits at the heart of decision-making, your prompts must continuously evolve alongside user preferences.

The AI Application Flood: Where It All Started

Since large language models went mainstream, job application volumes have exploded. The reason is straightforward — AI dramatically lowers the barrier for job seekers. Mayhew shared a striking data point: one of their clients received 2,740 applications within 24 hours of posting a single role. This is especially common for remote and entry-level positions.

Another stat he loves to cite: over the past two years, the average length of answers to questions like "Why do you want to join us?" has grown by roughly 50%. This clearly isn't because job seekers collectively became better writers — it's because LLMs are writing for them. Lower application friction translates directly into more work for recruiters.

This has a clear economic explanation. Before GPT-3.5/4 became widespread, writing a tailored cover letter took 30–60 minutes on average; today it takes under 5 minutes. When the marginal cost of an action approaches zero, rational actors dramatically increase their frequency of doing it — economists call this the "Jevons Paradox" playing out in digital labor markets: technology improves efficiency, but the total workload on the hiring side actually surges.

Recruiters need help — that's the problem Metaview set out to solve.

The Core Insight: User Preferences Will Always Evolve

Like any competent product team, Metaview started by interviewing stakeholders. Hiring managers would give classic requirements like "five years of backend experience," so you'd build an evaluation system around that.

But problems surfaced almost immediately. After reviewing the first batch of resumes, the hiring manager says, "Actually, I want someone with startup experience." You go back and rewrite everything. Then during the first interview, they say, "No, no, this candidate has never built something from 0 to 1 — that's a hard requirement." A requirement that simply didn't exist two days ago.

From this, Mayhew distilled what he considers the most important principle of the entire talk:

In any system where user judgment is central, user preferences will continuously evolve. People change their minds. If you want users to remain at the center of decision-making, your system must evolve with them.

He emphasized that this cannot be bolted on as an afterthought — it has to be the foundation. Prompts must reflect the reality that "preferences change" from day one of design.

System Architecture: Workflow + Agent Layer

Metaview's processing pipeline breaks down into a few stages. When a candidate enters the system, the first step is anonymization — removing names, emails, phone numbers, and other personally identifiable information so evaluations are based solely on experience, skills, and qualifications.

This is not just an engineering choice; it's a compliance measure against legal risks around hiring discrimination. Under the EU's GDPR, the U.S. EEOC (Equal Employment Opportunity Commission), and China's Personal Information Protection Law, hiring decisions based on name, gender, or race can be illegal. Academic "blind review" practices have demonstrated that gender bias drops significantly when reviewers don't know the author's identity. That said, anonymization isn't foolproof — details like alma mater or work location can still act as proxy variables that introduce hidden bias, which is why AI recruiting systems continue to face scrutiny from researchers.

The anonymized candidate profile is then matched against an Ideal Candidate Profile (ICP) to produce an evaluation.

The ICP concept is borrowed directly from the B2B sales world (where it stands for Ideal Customer Profile) — a strategic framework SaaS companies use to precisely define their target customer characteristics. Transplanted into recruiting, the core logic is the same: rather than passively screening through a flood of candidates, proactively define the characteristics of a "perfect fit." Unlike traditional job descriptions (JDs) that are written for legal compliance, the ICP focuses on implicit requirements — things like cultural fit, growth trajectory, and ways of thinking that are hard to quantify. The ICP is the part of the system that self-learns and self-improves.

How does the system learn? Since all pass/reject decisions are made by users, Metaview runs an Agent on top of the evaluation pipeline that watches for behavioral patterns — every pass, every rejection, every piece of feedback, every manual edit to the ICP is fed in as a "user message" to continuously optimize the ICP.

Why Workflows Instead of a Pure Agent?

Mayhew directly addressed the argument that "everything should be handed to a single agent." He acknowledged that developers naturally want to throw everything at one "maximalist agent," but business realities don't allow it.

When you're processing thousands of applications per day — and companies like Google and Anthropic receive hundreds of thousands to millions of applications per year — token costs become a hard constraint. You simply can't spend tens of dollars evaluating 3,000 applications for a single role. So Metaview's approach is: use efficient workflows at the base layer for high-volume evaluation, and use an Agent at the top layer for learning pass/reject patterns.

This architecture reflects an important design philosophy in LLM application engineering: "Use workflows for deterministic tasks, use agents for exploratory tasks." Workflows are predefined, predictable execution paths suited for high-frequency, low-variability tasks; Agents have tool-calling, dynamic decision-making, and multi-step reasoning capabilities suited for open-ended judgment tasks. This closely mirrors the "hot path / cold path" optimization approach in software engineering — push computationally expensive logic off the high-frequency execution path. Anthropic's official documentation also recommends: "When a workflow can solve the problem, there's no need to introduce the uncertainty of an Agent."

This layered thinking also shows up in model selection: Haiku for bulk candidate evaluation (cost-sensitive, latency-sensitive), Sonnet for unconstrained pattern-finding tasks (latency less important, stronger reasoning needed).

The three-tier Claude model family is designed precisely to accommodate different intelligence-cost-speed tradeoffs. Haiku's API call cost is roughly 1/5 that of Sonnet, making it ideal for high-concurrency, second-level-response scenarios; Sonnet is stronger at complex reasoning and tool calling, but significantly more expensive. This selection logic is essentially a variation of "multi-level caching" thinking: use cheaper models for standardized, repetitive "L1 tasks," and use expensive models for sparsely occurring "L2 tasks" — a similar pattern appears in RAG (Retrieval-Augmented Generation) system architectures.

A Key Tool: Query Files

Early on, the team noticed that user feedback tends to be relative. For example, "this person doesn't have enough Python experience" — but what does "enough" actually mean? The Agent needs the right context to understand it. They tried standard approaches like Bash and grep, but unstructured resume data is difficult to query directly.

So they built a dedicated tool called Query Files that traverses anonymized candidate resumes, understands the true meaning behind relative feedback, and feeds that context to the Agent responsible for maintaining the ICP.

Use Prose, Not Rules

Metaview explicitly rejects the use of weights, if-statements, or flowcharts. The longstanding criticism of keyword-matching resume screening is justified — that's simply not how you evaluate a person. Their approach is to fully leverage what LLMs do best — natural language reasoning — and let the model reason in prose, not flowcharts.

The Ideal Candidate Profile is a Markdown document structured around "role summary + must-haves + nice-to-haves + red flags." The reason for this structure: recruiters already think this way — the system faithfully mirrors the user's mental model rather than forcing users to adapt to weights and rules.

In a live demo, Mayhew gave feedback on a candidate from Airbnb: "Airbnb has a strong engineering culture — candidates from top engineering companies are a great fit." After submitting, Sonnet was invoked, the Agent analyzed the feedback and decided to call the tool to update the ICP, ultimately generating a new version with green/red diffs.

Interestingly, in production a single piece of feedback wouldn't trigger an ICP update. Instead, the system accumulates 100–200 feedback signals, identifies a pattern, and then makes adjustments — that's the right approach at scale.

Why Claude: You Need a Model That Can Push Back

Mayhew pointed out a particular challenge in resume evaluation: resumes are full of embellishment, and many candidates exaggerate their accomplishments. If you use an overly sycophantic model, it will take everything at face value and return inflated assessments.

"Sycophancy" is an important topic in LLM alignment research — the tendency for a model to give users the answer they want to hear rather than an objectively accurate one. Its root cause lies in biased reward signals during RLHF (Reinforcement Learning from Human Feedback) training: human annotators tend to rate "pleasing" responses more highly regardless of accuracy. In resume evaluation, a sycophantic model will uncritically accept self-promotion like "led a 5-person team to complete the most important project in company history." Anthropic has explicitly placed "honesty" above "pleasing the user" in Claude's training objectives — which is a key technical reason for choosing the Claude family. In high-stakes scenarios like financial crime detection and medical assessment, sycophancy is particularly dangerous.

Additionally, Anthropic provided them with special per-token input limits, making it feasible to process massive volumes of applications daily.

Three Core Takeaways

Mayhew closed by distilling the entire talk into three points:

1. User preferences will always evolve: Any evaluation system centered on user judgment cannot assume requirements are fixed at the start. Treat "preferences change" as the foundation of the system, not an afterthought patch.

2. Use prose, not rules: Embrace Markdown and natural language, and let Agents do what they do best. Don't regress to the flowcharts and if-statements of the pre-LLM era.

3. Build guardrails into the architecture: As LLMs are deployed in high-stakes domains like code review, financial crime detection, and KYC, evaluation systems are increasingly part of daily workflows. Guardrails cannot be added after the fact.

The philosophy running through all of this is "Human in the Center" — which is fundamentally different from the control theory concept of "Human in the Loop." The latter presupposes that humans are exception handlers: the system runs automatically by default, and humans only intervene when something goes wrong. "Human in the Center" is a philosophical shift: humans are not the safety net, they are the source of value and the decision-making authority. AI's role is demoted from "automation replacement" to "cognitive amplifier." This idea is in line with MIT Media Lab's tradition of "Intelligence Augmentation" and closely aligned with the EU AI Act's regulatory requirement for "meaningful human oversight" of high-risk AI systems.

The system plays the role of an apprentice: it does the grunt work of combing through thousands of applications, can flag whether a tech stack matches or whether a candidate has worked at a target company — but the final decision always belongs to the user.

Let the user be the master. Let the system be the apprentice.

This philosophy offers a valuable reference point for any team building AI evaluation systems: while pursuing automation efficiency, designing a system that can self-evolve yet always respects human judgment will be a defining challenge in AI application engineering.

Key Takeaways