FDE-Gym
A benchmark for the customer delivery capabilities of AI agents, comprising end-to-end delivery tasks drawn from 18 industries and 20 customer scenarios. It simulates the real working conditions of a forward-deployed engineer facing ambiguous requirements and production environments, and evaluates an agent's ability to proactively clarify requirements and incrementally modify systems.
Timeline (last 90 days)
The FDE-Gym benchmark covers 95 end-to-end delivery tasks across 18 industries and 20 customer scenarios.
On FDE-Gym, the best configuration achieved an average score of only 0.56, solving only 14 of the 95 tasks.
On the 23 FDE-Gym tasks where all key facts had been obtained, the best configuration achieved meaningful business improvement on only 10 tasks.
Discovering implicit constraints is the primary obstacle for agents in real customer delivery scenarios.
In the setting closest to reality (where unstated constraints must be actively discovered), the best-performing model configuration achieves an average FDE-Gym score of only 0.56.
Among the 95 tasks, only 14 tasks were truly solved by the best configuration (score no lower than 0.8 and no incidents).
In 23 tasks, the best configuration successfully obtained all key facts from stakeholders, but among them only 10 tasks achieved a normalized business improvement of 0.8 or above.
FDE-Gym constructs 95 end-to-end delivery tasks from 18 industries and 20 customer scenarios, each task adapted by industry experts from real delivery projects.
Information discovery capability alone cannot explain the entire gap: discovering constraints is a necessary but not sufficient condition for delivery success.
FDE-Gym's penalty mechanism shows that in actual delivery, "asking the right questions" is more important than "asking many questions," requiring the Agent to have the ability to estimate the value of information.
5 more timeline events
All Facts (15)
The FDE-Gym benchmark covers 95 end-to-end delivery tasks across 18 industries and 20 customer scenarios.
50%UnverifiedOn FDE-Gym, the best configuration achieved an average score of only 0.56, solving only 14 of the 95 tasks.
50%UnverifiedOn the 23 FDE-Gym tasks where all key facts had been obtained, the best configuration achieved meaningful business improvement on only 10 tasks.
50%UnverifiedDiscovering implicit constraints is the primary obstacle for agents in real customer delivery scenarios.
50%UnverifiedIn the setting closest to reality (where unstated constraints must be actively discovered), the best-performing model configuration achieves an average FDE-Gym score of only 0.56.
50%UnverifiedAmong the 95 tasks, only 14 tasks were truly solved by the best configuration (score no lower than 0.8 and no incidents).
50%UnverifiedIn 23 tasks, the best configuration successfully obtained all key facts from stakeholders, but among them only 10 tasks achieved a normalized business improvement of 0.8 or above.
50%UnverifiedFDE-Gym constructs 95 end-to-end delivery tasks from 18 industries and 20 customer scenarios, each task adapted by industry experts from real delivery projects.
50%UnverifiedInformation discovery capability alone cannot explain the entire gap: discovering constraints is a necessary but not sufficient condition for delivery success.
50%UnverifiedFDE-Gym's penalty mechanism shows that in actual delivery, "asking the right questions" is more important than "asking many questions," requiring the Agent to have the ability to estimate the value of information.
50%UnverifiedFDE-Gym's experiments show that even when the Agent grasps all key facts, the conversion success rate is still less than half, indicating that current models have systematic shortcomings in planning and execution capability on long-chain complex tasks.
50%UnverifiedWhen the model needs to discover unstated constraints on its own without explicit guidance, performance drops sharply.
50%UnverifiedFDE-Gym's differentiated positioning lies in that the task input itself is incomplete, the Agent must first complete the information through proactive interaction, and the evaluation metric is the open-ended degree of business KPI improvement rather than a right-or-wrong judgment.
50%UnverifiedFDE-Gym's scoring mechanism is normalized over the interval from 0 (system not improved) to 1 (effect of the reference solution), and penalizes interaction behaviors with stakeholders (sending messages, trial runs, waiting time).
50%UnverifiedThe FDE-Gym platform provides an online human practice mode using the same scoring standard, providing a baseline for human-versus-AI comparison.
50%