Can LLM Agents Work as Forward-Deployed Engineers? An Enterprise Text-to-SQL Reality Check

Introduction: From Coding Assistant to "Forward-Deployed Engineer"

LLM coding agents are reshaping how software is delivered—but can they go further and, like a Forward-Deployed Engineer (FDE), dive into customer contexts, uncover ambiguous requirements, and deliver usable production systems? An empirical study on enterprise-grade Text-to-SQL tasks brings this question to the forefront.

The study built a simulated customer whose requirements were designed around a real enterprise Text-to-SQL use case, with the initial request deliberately kept vague. The agent first had to clarify acceptance criteria, data permissions, and hardware constraints during a "discovery phase," then plan and execute a post-training approach during a "delivery phase." The results are not encouraging, but they are highly instructive.

Forward-Deployed Engineer (FDE) refers to an engineering role that sits on-site with the customer and is responsible for deeply integrating a product's capabilities with the customer's actual business scenarios—common at enterprise software companies such as Palantir and Databricks. The core value of an FDE is not writing generic code, but rather: communicating repeatedly with the customer to clarify vague or even contradictory requirements, understanding the customer's data permissions and compliance boundaries, and designing and delivering a runnable system under real hardware and budget constraints. Mapping this role onto LLM agents means the agent must not only write code but also proactively ask follow-up questions, cross-verify, plan post-training experiments, and self-check against multidimensional acceptance criteria. This paper uses enterprise-grade Text-to-SQL tasks as its testing ground to examine whether an agent can end-to-end assume this dual responsibility of "requirement discovery + solution delivery."

Discovery Phase: The Strongest Models Can Uncover 80% of Hidden Requirements

In the discovery phase, the study tested 15 models. The strongest performers could uncover more than 80% of hidden requirements, while open-weight models uncovered only 40–50%. This means leading models already have some practical value in requirement elicitation, but a clear gap remains between top closed-source models and open-source models.

More importantly, the discovery phase is not a one-shot Q&A; it requires the agent to proactively ask follow-up questions and cross-verify. The simulated customer in the study does not volunteer all constraints, so the agent must, like a real FDE, progressively approach the true requirements through dialogue. This capability currently depends heavily on the model's reasoning and planning level.

Delivery Phase: Only a Quarter of Configurations Pass All Acceptance Criteria

The delivery phase poses an even greater challenge. Among eight coding-agent configurations, only 25% of runs passed all acceptance criteria. In other words, even when requirements have been correctly discovered, turning the solution into a runnable, acceptable deliverable remains a weak spot for most agents.

The study notes that the delivery phase requires the agent to complete data preparation, model selection, post-training recipe design, and final evaluation. Each step can deviate, and these deviations get amplified in subsequent stages. Acceptance criteria often span multiple dimensions, and failing any one of them causes overall failure.

Post-training refers to the process of adapting a pretrained foundation model to a specific task and domain data through continued pretraining, supervised fine-tuning (SFT), preference alignment (such as RLHF/DPO), and similar methods. In enterprise Text-to-SQL scenarios, a post-training recipe typically must decide: which foundation model to use, how many labeled SQL samples to use, whether to introduce schema linking and retrieval augmentation, which fine-tuning strategy and hyperparameters to adopt, and how to evaluate on a held-out set. The recipe design space is large, and different choices significantly affect final accuracy and deployment cost—which is why the study emphasizes that the "lack of systematic comparison of foundation models and post-training methods" becomes a bottleneck: if the agent cannot systematically traverse and record these configurations, it is hard to judge which solution is truly suitable for production.

Four Major Bottlenecks: Incomplete Discovery, Unreliable Execution, Wasted Compute, and Benchmark Distortion

The study identifies four core bottlenecks. First, incomplete discovery: 40% of hidden requirements are missing or misrepresented in the final solution. Second, unreliable execution: the agent's operational stability during the delivery phase is insufficient. Third, insufficient utilization of the compute budget, along with a lack of systematic comparison of foundation models and post-training methods. Fourth, benchmark accuracy does not guarantee production readiness—there is a gap between a model's high score on an evaluation set and its actual deployment performance.

These four bottlenecks are interrelated. Incomplete discovery leads subsequent experiments astray, unreliable execution wastes compute, and the lack of systematic comparison makes it hard for teams to judge which solution is truly suitable for production. Benchmark distortion can make teams overly optimistic too early and cause them to overlook edge conditions in real scenarios.

Benchmark-reality gap refers to the systematic discrepancy between a model's score on public evaluation sets and its performance in real production environments. Typical benchmarks in the Text-to-SQL field (such as Spider and BIRD) usually provide clean schemas, well-defined single-turn questions, and standard answers, whereas real enterprise scenarios often involve dirty data, multi-turn clarification, permission filtering, cross-database joins, and ambiguity in metric definitions. A model may achieve high scores by memorizing benchmark patterns or exploiting distributional features of the evaluation set, yet fail when faced with real data permissions and hardware constraints. The study therefore recommends not using benchmark scores as the sole basis for model selection, but rather conducting small-scale validation under real data permissions and hardware conditions, treating benchmark results as a necessary but not sufficient condition.

Paths for Improvement: Human Review, Process Monitoring, and Experiment Standards

To address these issues, the study offers three recommendations. First, conduct human review of the requirements discovered by the agent, especially regarding data permissions and compliance constraints. Second, monitor the agent's plan during delivery execution rather than looking only at the final result. Third, establish clear standards for the experimental environment, covering hardware, budget ranges, and production checklists.

These recommendations point to a common conclusion: current LLM agents are not yet capable of independently taking on the FDE role, but they can serve as efficient copilots for human engineers. Handing discovery-phase outputs to humans for confirmation, exposing the delivery-phase process to monitoring, and clearly defining experimental boundaries can significantly reduce risk.

Implications for Enterprises

If an enterprise wants to try using LLM agents for Text-to-SQL-type requirements, it should prioritize models with strong discovery capabilities and set human checkpoints for the delivery phase. At the same time, it should not use benchmark scores as the sole basis for model selection, but rather conduct small-scale validation under real data permissions and hardware conditions. FDE agents will take time to mature, but this study has already outlined a clear path for improvement.

One-line Summary

An empirical study shows LLM agents are not yet ready to independently serve as forward-deployed engineers, with clear shortcomings in both discovery and delivery.

Paragraph Summary

This article introduces an empirical study that places LLM coding agents in enterprise-grade Text-to-SQL scenarios and simulates the role of a forward-deployed engineer (FDE). The study built a simulated customer with initially vague requirements, requiring the agent to first clarify acceptance criteria, data permissions, and hardware constraints before planning and executing a post-training approach. Results show that in the discovery phase, the strongest models identified over 80% of hidden requirements, while open-weight models managed only 40–50%; in the delivery phase, only 25% of eight configurations passed all acceptance criteria. The study identifies four major bottlenecks—incomplete discovery, unreliable execution, wasted compute, and benchmark distortion—and proposes three improvements: human review of requirements, monitoring of execution plans, and clear experimental environment standards. The conclusion is that current agents are better suited as copilots for human engineers than as independent FDEs.