Evals and RL Environments: The Critical Challenges of Deploying AI in the Real World

A founder dinner conversation featuring Vincent Sun Chen from Snorkel AI brought some of the hottest — and most intractable — topics in the AI industry to the table: model evaluation (evals) and reinforcement learning (RL) environments. In an era where model capabilities are shattering benchmarks at a breathtaking pace, how we measure intelligence and how we train truly reliable models are becoming the defining factors in whether AI can actually be deployed in production.

The Cold Reality Behind the Evals Hype

Over the past few years, companies like Snorkel — focused on data and RL environments — have experienced explosive growth, and market interest in evaluation has never been higher. The reason is straightforward: with every new model release, existing benchmarks get crushed. When benchmarks are broken this quickly, the old yardsticks stop working, and the industry urgently needs new ways to understand where a model is strong and where it falls short.

This sets up the central questions of the conversation: which evals have already been well-defined by the community, which remain unsolved, and which capabilities should be left to frontier models versus which ones should represent intelligence you genuinely own and control. This isn't just a technical question — it goes to the heart of what a company's moat looks like in the age of AI.

Three Core Challenges in Building RL Environments

Fairness: Who's Actually to Blame for Failure?

One of the biggest challenges in building RL environments is fairness. When a model fails in a given environment, where does the problem actually lie — in the input data, the execution harness, or the reward model itself? This difficulty in attribution directly impacts training quality. If you can't accurately pinpoint the root cause of failure, your optimization efforts may be pointed in entirely the wrong direction.

Reward Design: The Difficulty of Quantification and the Risk of Gaming

Designing sound reward mechanisms is equally tricky. Many tasks are inherently hard to quantify — you can't simply assign a score to capture whether something was done well. At the same time, you have to guard against reward hacking, where the model finds loopholes to score well without actually completing the task. Even more subtle is the constraint that intermediate rewards can't be designed to be too rigid or prescriptive, or they'll constrain the model's exploration space. Reward design becomes an art of balancing constraint and freedom.

What is reward hacking? Reward hacking is a classic problem in reinforcement learning, where a model optimizes the loopholes in a reward function rather than genuinely completing the task. Classic examples include a game-playing AI learning to spin in circles to accumulate points rather than finishing the track, or a robotic arm blocking the camera to avoid penalty signals. In the context of LLM RLHF, the problem is even more subtle — models may learn to generate responses that sound fluent and please human evaluators but are actually inaccurate. Intermediate reward design is equally delicate: rewards that are too sparse make learning difficult, while rewards that are too dense or specific create a "shortcut effect" that limits how freely the model explores the solution space. This is why, in complex business tasks, reward engineering often consumes more iteration cycles than model architecture itself.

Long-Horizon Evaluation: The Feedback Problem That Takes Weeks or Months

Long-horizon evaluation remains extremely difficult to crack. Some real-world business processes may take weeks or even months to produce a final outcome. At that timescale, providing effective feedback signals to a model is a problem that current techniques are far from solving.

"Good Enough" Is Far From Good Enough in the Real World

In regulated industries, most use cases still require human in the loop to guarantee accuracy approaching 100%. One pointed observation from the conversation: "80% accuracy is not good enough." For high-stakes domains like finance and healthcare, this means full automation is unlikely to completely replace human review in the near term.

A related phenomenon is jagged intelligence — models perform brilliantly on many tasks, yet still stumble on long-tail edge cases. This uneven distribution of capability is a risk that enterprises must confront head-on when deploying AI.

What is jagged intelligence? The concept of "jagged intelligence" was coined to describe the highly uneven capability profile of large language models — a model can outperform most humans on the bar exam, yet frequently fail at simple spatial reasoning or counting tasks. Compared to a human capability curve, a model's capability boundary is not a smooth gradient but an unpredictable "jagged" shape. This creates a dual challenge for enterprise AI deployment: on one hand, strong performance on hard benchmarks doesn't justify overconfidence in overall reliability; on the other hand, model failures often appear in places that human intuition would consider "easy," making it nearly impossible to exhaustively cover all failure modes in testing. This is why a systematic evaluation framework — rather than a single benchmark score — is increasingly seen as the core tool for assessing a model's real-world usability.

The "Barbell Structure" of Data Supply

As models become increasingly capable, the data supply market may evolve into a barbell structure: at one end, premium niche data vendors (think specialized data serving SMBs), and at the other end, large-scale data providers — with the middle squeezed out.

The underlying logic is this: collecting unique data for specific tasks and then post-training on it often yields higher accuracy at lower cost. This means that even as frontier model capabilities continue to grow, the intelligence you own — capabilities trained on proprietary data — retains long-term value. Whoever controls unique data may be able to build a hard-to-replicate advantage in vertical domains.

What is post-training? Post-training broadly refers to any further training performed after a foundation model's pre-training is complete, typically including supervised fine-tuning (SFT), reinforcement learning alignment (e.g., RLHF), and domain adaptation using proprietary data. Compared to training a large model from scratch, post-training requires a fraction of the compute but can deliver significant accuracy improvements in vertical use cases. This amplifies the value proposition of "owning unique data" in the post-training era — even without the resources to train a frontier foundation model, companies can post-train on proprietary data to build customized capabilities that outperform general-purpose LLMs on specific tasks. The barbell structure in the data supply market is the commercial reflection of this logic: vendors who can supply high-quality domain-specific data, and large platforms capable of processing general-purpose data at scale, will occupy the two ends — while commoditized mid-tier data providers face pressure to be marginalized.

Closing Thoughts

This dinner was the third installment in the founder dinner series. It surfaces an emerging consensus: beneath all the noise of benchmark-chasing, the real competition lies in evaluation frameworks, reward design, and control over proprietary data. Drawing the line between general intelligence and proprietary intelligence may well be the most important question in AI commercialization over the next several years.