Ankur
A technical speaker in the field of AI engineering, focused on debugging agent production environments and building evaluation systems.
Timeline (last 90 days)
An on-site survey shows that the vast majority of people validate agents by gut feeling, few have a configured evaluation system, and very few believe their evaluation coverage is sufficient to iterate with confidence.
The standard agent improvement process includes: discover a production problem → find the corresponding trace → sample to form a dataset → analyze the root cause → design a solution → build an evaluation → run experiments → ship, then loop.
Bringing trace data, existing evaluation results, human annotations, APM monitoring logs, and the codebase together in front of a coding agent (such as Claude Code, Cursor, Codex) can automate most of the investigation work, compressing a process that originally took days into minutes.
In the Korean stock query case, the baseline pass rate was 0%, normalizing country names in the first round was ineffective, and after updating the tool version, the pass rate rose to about 90% in the third iteration.
The core value of evaluation is preventing performance regressions, because prompts may degrade due to model updates or changes in how users phrase questions, and without evaluation such degradation cannot be perceived.
All Facts (5)
An on-site survey shows that the vast majority of people validate agents by gut feeling, few have a configured evaluation system, and very few believe their evaluation coverage is sufficient to iterate with confidence.
50%UnverifiedThe standard agent improvement process includes: discover a production problem → find the corresponding trace → sample to form a dataset → analyze the root cause → design a solution → build an evaluation → run experiments → ship, then loop.
50%UnverifiedBringing trace data, existing evaluation results, human annotations, APM monitoring logs, and the codebase together in front of a coding agent (such as Claude Code, Cursor, Codex) can automate most of the investigation work, compressing a process that originally took days into minutes.
50%UnverifiedThe core value of evaluation is preventing performance regressions, because prompts may degrade due to model updates or changes in how users phrase questions, and without evaluation such degradation cannot be perceived.
50%UnverifiedIn the Korean stock query case, the baseline pass rate was 0%, normalizing country names in the first round was ineffective, and after updating the tool version, the pass rate rose to about 90% in the third iteration.
50%