FDE-Gym Benchmark: Why the Last Mile of AI Delivery Gets Stuck

A New Battleground for AI Agent Evaluation: From Well-Defined Tasks to Ambiguous Requirements

Most existing Agent benchmarks focus on well-defined tasks—issue reports, specification documents, task tickets. These evaluations share a common assumption: the problem has already been fully described, and the Agent merely needs to execute.

But real-world client delivery is entirely different. Forward-Deployed Engineers (FDEs) face ambiguous requirements, live production systems, and a large number of unstated constraints. FDE-Gym was proposed precisely to fill this evaluation gap: it constructs 95 end-to-end delivery tasks drawn from 20 client scenarios across 18 industries, with each task adapted by domain experts from real delivery projects.

Forward-Deployed Engineer (FDE) is an engineering role originating from enterprise software companies like Palantir, whose core characteristic is being directly embedded at the client site, responsible for adapting general product capabilities to the client's specific business scenarios. FDEs do not face clear requirements documents, but rather vague verbal descriptions from clients, the historical baggage of legacy systems, and unspoken political constraints within the organization. This role's working pattern closely mirrors the positioning of AI Agents in real delivery: Agents need to proactively clarify requirements, make incremental modifications on existing systems, and deliver measurable business value within limited time. FDE-Gym builds its evaluation around this role as a template, essentially using "can AI work like an excellent FDE" as the measuring stick.

Agent Benchmarks have been a rapidly developing evaluation field in recent years, with representative works including SWE-bench (real GitHub issue fixing), WebArena (web operation), GAIA (general assistant tasks), and others. These benchmarks share the common trait of relatively complete task descriptions and clear scoring criteria (e.g., whether tests pass, whether answers match). FDE-Gym's differentiated positioning lies in: the task input itself is incomplete, the Agent must first proactively interact to fill in information, and the evaluation metric is the open-ended degree of business KPI improvement rather than simple right-or-wrong judgment. This makes it closer to the complexity of real client delivery.

Evaluation Design: Simulating a Real Delivery Environment

FDE-Gym's core innovation lies in the paradigm shift in evaluation. It does not have the Agent answer a question or generate a piece of code, but rather has the Agent interact with scripted stakeholders in a simulated client environment, operate on modifiable production-like systems, and ultimately deliver a usable solution.

The scoring mechanism revolves around the client's own success metrics: normalized on a scale from 0 (system unimproved) to 1 (reference solution). Meanwhile, interactions with stakeholders (sending messages, trial runs, wait times) are all counted as penalties—meaning the Agent cannot "grind out" an answer through unlimited communication but must weigh the cost of information acquisition like a real FDE.

The platform also provides an online human practice mode using the same scoring criteria, providing a baseline for human-vs-AI comparison.

Scripted stakeholders are key components in the FDE-Gym environment, referring to virtual client roles driven by predefined rules or language models, such as business owners, operations engineers, data analysts, etc. The Agent can ask them questions or request data by sending messages, but each interaction incurs a cost penalty. This design simulates the dilemma FDEs face in reality: client time is limited, communication bandwidth is scarce, and frequent interruptions damage trust. Unlike open-ended dialogue environments, scripted stakeholders' response scope and trigger conditions are strictly controlled to ensure evaluation reproducibility.

The normalized scoring (0 to 1 scale) design allows scores across different tasks to be compared horizontally. A score of 0 corresponds to the Agent making no effective improvement to the system, while 1 corresponds to the effect achieved by the reference solution provided by the team. This relative scoring avoids the industry-difference problem of absolute metrics—for example, e-commerce conversion rate improvement and manufacturing yield improvement cannot be directly compared, but after normalization both map to the same scale. It should be emphasized that scoring also penalizes meaningless interaction behavior, meaning the Agent's optimal strategy is not to maximize information gathering but to find a balance between information sufficiency and interaction cost.

Report Card of 13 Model Configurations: The Best Only Scores 0.56

The research team evaluated 13 model configurations from 7 providers. In the setting closest to reality—where unstated constraints need to be proactively discovered—the best-performing configuration achieved an average FDE-Gym score of only 0.56, with only 14 out of 95 tasks truly resolved (score no less than 0.8 and no incidents).

This number means: even the current strongest model configuration cannot deliver qualified results in over 85% of real delivery scenarios.

Bottleneck Analysis: Discovering Implicit Constraints Is the First Hurdle

The research reveals a clear bottleneck: when models need to discover unstated constraints on their own without explicit guidance, performance drops sharply. This stands in stark contrast to model performance on well-defined tasks in traditional benchmarks.

But this is only half the story. The team found that information discovery capability alone cannot explain the entire gap. In 23 tasks, the best configuration successfully obtained all key facts from stakeholders, but only 10 of these tasks achieved normalized business improvement above 0.8.

This shows the problem is not just about "finding information," but also about "how to translate the information into an effective delivery solution once obtained." Discovering constraints is a necessary condition, but far from sufficient.

Unstated constraints refer to conditions not explicitly stated by the client in the requirements description but that constitute hard limits on solution feasibility. For example, a client may request "optimize the recommendation system" but not mention that the system must run without additional GPU resources, or cannot use certain types of user data (for compliance reasons). In FDE-Gym's evaluation, such constraints are intentionally hidden in stakeholder conversations or system behavior, and the Agent must proactively discover them through questioning, trial runs, observing anomalies, and other means. This capability is almost never tested in traditional benchmarks because those tasks typically already include all constraints in the description.

The difference between task resolution rate and normalized score is worth clarifying. The normalized score measures "how much was improved," while the task resolution criterion (score ≥ 0.8 and no incidents) requires the Agent to both achieve results close to the reference solution and avoid introducing new system failures or violating constraints. The inclusion of the "no incidents" condition reflects the reality that in production environments, "not causing damage" is as important as "making improvements." This also explains why there is a significant gap between the score of 0.56 and the resolution rate of 14/95: for some tasks, the Agent may have achieved some improvement but was not counted as resolved due to triggering incidents or failing to meet the threshold.

Implications for AI Delivery

FDE-Gym's results point to a fact long underestimated by the industry: there is a substantial gap between solving well-defined tasks and delivering effective AI systems for clients.

For teams building AI Agent products, this means several directions deserve attention: first, the Agent's proactive information acquisition capability needs to be optimized under cost constraints rather than explored at any cost; second, even with complete information, there remain many engineering problems to solve in the translation chain from understanding to delivery; third, FDE-Gym provides a quantifiable evaluation framework that makes progress on the "last mile" measurable.

The last step of AI delivery may be harder than all the preceding steps combined.

Proactive information acquisition under cost constraints is a new direction in Agent capability research. Traditional frameworks like ReAct and Reflexion encourage Agents to explore and reflect as much as possible, but FDE-Gym's penalty mechanism shows that in real delivery, "asking the right questions" matters more than "asking many questions." This requires the Agent to have the ability to estimate information value: judging whether a given interaction is likely to lead to the discovery of key constraints, and the weight of that discovery's impact on the final solution. This capability has not yet been effectively trained in large models and is a direction worth investing in for future Agent research.

The translation chain from understanding to delivery involves multiple engineering steps: constraint formalization, solution design, code implementation, system integration, validation and rollback, etc. FDE-Gym's experiments show that even when the Agent possesses all key facts, the translation success rate remains below half, indicating that current models have systematic shortcomings in planning and execution for long-chain complex tasks. This finding has cautionary significance for enterprises deploying AI Agents: do not assume that "sufficient information means delivery," but retain human judgment and fallback mechanisms in the human-AI collaboration workflow.

One-line Summary

FDE-Gym evaluation shows the strongest AI Agent achieves an average score of only 0.56 in real client delivery scenarios, revealing a substantial gap between well-defined tasks and ambiguous requirements.

Paragraph Summary

FDE-Gym is an AI Agent evaluation benchmark for real client delivery scenarios, containing 95 end-to-end tasks from 20 client scenarios across 18 industries. Unlike traditional benchmarks, it requires Agents to proactively discover unstated constraints under ambiguous requirements, operate on production-like systems, and interact with scripted stakeholders, with interaction costs counted as penalties. The study evaluated 13 model configurations from 7 providers, with the best configuration averaging only 0.56 and only 14 out of 95 tasks truly resolved. Bottleneck analysis shows that discovering implicit constraints is the first hurdle, but even with all key information obtained, the delivery translation rate remains unsatisfactory. This work reveals a substantial capability gap between solving well-defined tasks and delivering effective AI systems for clients, pointing Agent product teams toward directions worth attention such as proactive information acquisition, cost-constrained optimization, and the delivery translation chain.