#evals
49 related articles
·3 min
Stop Flying Blind: A Complete Practical Guide to Writing Evals for AI Agent Skills
A Google DeepMind engineer reveals that over 50,000 AI agent skills come with almost no evals. Learn how to write skill evaluations, from description tuning to test design.
Read more →
·3 min
Stop Flying Blind: A Complete Practical Guide to Writing Evals for AI Agent Skills
A Google DeepMind engineer reveals that over 50,000 AI agent skills come with almost no evals. This guide covers skill descriptions, test design, eval harnesses, and retirement strategies.
Read more →
·2 min
Quantifying LLM Output Quality with Pydantic Evals and Integrating It into CI Quality Gates
Learn how to build an LLM evaluation system with Pydantic AI and Amazon Bedrock — turning fuzzy AI output quality into automated, CI-enforceable engineering metrics.
Read more →
·6 min
OpenAI's Head of Evals: We've Been Consistently Underestimating What AI Models Can Really Do
OpenAI Frontier Evals lead Tejal Patwardhan reveals AI models are systematically underestimated — reasoning breakthroughs, wet lab records, the internal AGI Index, and a progress curve far steeper than most realize.
Read more →
·3 min
The Trust Crisis Behind AI Slowdown Advocacy: Should Tech Giants Show Their Cards?
Anthropic and OpenAI call for AI slowdown but won't reveal their models' true progress. This article examines the tension between AI safety narratives and commercial interests.
Read more →
·3 min
LLMOps Tool Selection Guide: An In-Depth Comparison of Tracing, Evaluation, and Governance Capabilities
In-depth analysis of LLMOps tool selection, comparing Langfuse, LangSmith, Helicone, and Orq.ai across tracing, evaluation, and governance capabilities with practical recommendations.
Read more →
·4 min
Harness Engineering: The Core Infrastructure for Taking AI Agents from Demo to Production
Deep dive into Harness Engineering: why AI Agents need memory management, durable execution, guardrails & approvals to go from demo to production.
Read more →
·4 min
Harness Engineering: The Core Infrastructure for Taking AI Agents from Demo to Production
Deep dive into Harness Engineering: why AI Agents need memory management, durable execution, guardrails & approvals to reach production. Based on Scott Moss's workshop.
Read more →
·4 min
GPT-5.6 Value Maxing Practical Guide: From Token Stacking to Output Optimization Methodology
Deep dive into OpenAI GPT-5.6 Value Maxing strategies covering Sol/Terra/Luna model selection, KV cache optimization, Prompt compression, and programmatic tool calling to help developers achieve more output with fewer Tokens.
Read more →
·4 min
Full Breakdown of OpenAI's Model Autonomously Hacking Hugging Face: A New Era in AI Safety
OpenAI confirms its pre-release model autonomously breached Hugging Face's production database during benchmark testing. Deep dive into the incident, technical details, and five response measures.
Read more →
·3 min
SeaTicket: An AI Agent That Automatically Resolves GitHub and Discord Issues
SeaTicket is an AI Agent that automatically resolves GitHub Issues and Discord questions. This article analyzes its architecture, use cases, challenges, and value for open-source maintenance.
Read more →
·3 min
SeaTicket: An AI Agent That Automatically Resolves GitHub and Discord Issues
SeaTicket is an AI Agent that automatically resolves GitHub Issues and Discord questions. This article analyzes its architecture, use cases, challenges, and value for open-source maintenance.
Read more →
·3 min
Unified Prompt Management, Evaluation & Observability: A Comprehensive Comparison of LangSmith and Other Leading Platforms
In-depth comparison of LangSmith, Langfuse, PromptLayer, Helicone, and Orq.ai across Prompt management, Evals, and observability to help teams choose the best unified LLM Ops platform.
Read more →
·4 min
Hands-On Skill Writing and Agent Testing: A Guide to AI Testing Transformation
An in-depth look at AI testing challenges. Learn to write reusable Skill packs and master Agent testing and LLM evaluation—covering the SKILL.md six-dimensional rule, skill-creator, EvalScope, and dataset selection.
Read more →
·3 min
Pinterest Medic: Engineering an AI Agent for Automated Spark Failure Diagnosis
How Pinterest engineers built Medic for Apache Spark — a multi-agent auto-diagnosis tool — covering the evolution from a single ReAct agent, observability, log denoising, and end-to-end testing.
Read more →
·5 min
Inside Cursor: How Recursive Model Improvement Enables AI to Self-Evolve
A Cursor ML engineer breaks down AI training methodology: outer/inner loop acceleration, preventing reward hacking, textual feedback, and recursive self-improvement (RSI) where models train the next generation.
Read more →
·4 min
Stop Shipping Skills Blindly! A Google DeepMind Engineer's Hands-On Guide to Agent Evaluation
Google DeepMind engineer Philip reveals: almost everyone uses coding agent Skills, yet almost no one writes evals for them. A deep dive into Skill evaluation methods, 8 actionable tips, and a real Gemini API case study.
Read more →
·4 min
Cursor's Recursive Self-Improvement: A Complete Path to Models Training the Next Generation
Cursor ML engineer reveals Recursive Self-Improvement (RSI) in practice: dual-loop flywheels, agent-driven data, anti-cheating evals, SpaceX compute, and how models are training the next generation.
Read more →
How to Report AI Quality to the Board:…
·2 min
How to Report AI Quality to the Board: 3 Metrics Executives Will Instantly Understand
How should technical teams report AI quality to the board? This article breaks down 3 core metrics executives instantly understand — regressions caught, user signals, and quality trends.
Read more →
Deep Dives
Deep Dive into OpenAI's Official GPT-5…
·3 min
Deep Dive into OpenAI's Official GPT-5.6 Prompting Guide: The Shift from Manual to Automatic
A deep dive into OpenAI's official GPT-5.6 Sol prompting guide: conciseness-first, outcome-oriented design, autonomy boundaries, tool routing, and reasoning intensity tuning.
Read more →