Sofia Jakovcic
A forward-deployed engineer at Arize, focusing on agent evaluation and regression testing practices.
Timeline (last 90 days)
Agents rarely fail at the code level; they fail at runtime.
After adding the prompt "confirm before performing any operation" to a customer service agent, production evaluation showed the "operation safety" metric improved by 17 points and "task completion" dropped by 17 points, with the average net value of the two metrics coming out to exactly zero.
All Facts (2)
Agents rarely fail at the code level; they fail at runtime.
50%UnverifiedAfter adding the prompt "confirm before performing any operation" to a customer service agent, production evaluation showed the "operation safety" metric improved by 17 points and "task completion" dropped by 17 points, with the average net value of the two metrics coming out to exactly zero.
50%