Spark Declarative Pipelines: Data Infrastructure Built for Agents

Data engineering is undergoing a quiet transfer of power: the ones writing pipelines are increasingly no longer data engineers, but AI agents. According to the original article, more than 70% of production SDP (Spark Declarative Pipelines) are now written by AI agents. If this figure is accurate, it means that an API surface designed for human engineers is being dominated by non-human users. The question naturally follows: when the primary users become agents, how should the pipeline framework itself be designed?

From Human-Readable to Agent-Writable

The traditional design assumption for data pipelines is "written by humans, read by humans, maintained by humans." Developers are familiar with imperative code and accustomed to using loops, conditionals, and exception handling to precisely control execution paths. But agents write pipelines differently: they are better at generating declarative goal descriptions under explicit constraints than at making local patches within lengthy procedural code.

This is precisely where the value of declarative pipelines lies. You declare "what this table should look like, which upstream sources it depends on, which data quality constraints it must satisfy," and the framework handles deriving the execution plan, managing dependency order, and retries. The core argument of the original article can be summarized in one sentence: the more derivation responsibility the framework bears, the less code the agent needs to explicitly generate, and the smaller the error surface. Human engineers gain simplicity from this; agents gain generability.

This also explains a design tension that is easily overlooked. Imperative APIs are human-friendly because humans can use a debugger to trace each step; but they are agent-unfriendly because each step is a decision point that must be correctly generated. The longer the pipeline, the greater the cumulative generation error. Declarative approaches compress this error surface down to the "goal description" layer.

The divide between declarative and imperative pipelines is essentially the separation of "what to do" from "how to do it." Imperative code specifies the execution path step by step—which table to read first, how many times to loop, how to catch exceptions; declarative code only describes the desired end state, e.g., "the target table should contain these columns, come from these upstream sources, and satisfy these constraints," and the framework's query planner derives the concrete execution plan. Spark Declarative Pipelines (SDP) is this paradigm built on top of Spark: it delegates dependency resolution, execution order, incremental refresh, and failure retries to the runtime, and developers (or agents) only need to declare dataset definitions and quality rules. This pattern can be traced back to the traditions of SQL and Makefile: declare goals and dependencies, and let the executor decide the steps. For agents, the declarative interface transforms the problem of "generating a correct program" into "generating a correct specification," and specifications are far more structured than free-form code, making them easier to statically check and automatically repair.

Why Agents Prefer Declarative

Generation Cost and Verification Cost

The cost structure of agent code-writing differs from that of humans. For humans, writing code is expensive in thinking and cheap in typing; for agents, generation is relatively cheap, and the real bottleneck is verification—whether the generated pipeline is correct, idempotent, and stable under data drift. Declarative pipelines delegate "how to execute" to the framework and leave "what result is needed" to the agent, converging the verification target from an entire block of procedural code into a checkable goal state. This makes automated verification feasible.

Idempotency is the cornerstone of data pipeline reliability: executing the same pipeline multiple times on the same input should produce consistent results without duplicate rows or side effects. In imperative pipelines, idempotency must be manually ensured by developers—for example, using upserts, deduplication keys, or transactional writes; agents generating such logic can easily miss edge cases. Declarative frameworks typically build idempotency into runtime semantics: the framework decides whether to fully recompute or incrementally merge based on the target table definition and upstream changes, and guarantees that retries do not pollute results. This directly reduces the agent's verification burden—it does not need to prove that every step it writes is idempotent, only that the declared goal state is correct. Similarly, data drift (changes in upstream schema or distribution) manifests under the declarative model as contract validation failures at the framework level, rather than hidden errors in procedural code, making problems surface earlier and easier to locate.

Constraints as Guardrails

Declarative frameworks naturally come with a set of constraints: dependencies must be explicitly declared, schemas must be defined, and data quality rules are first-class citizens. These constraints may be "troublesome" for humans, but they are guardrails for agents. Agents in spaces with too much freedom tend to generate code that looks reasonable but is actually fragile; structured constraints narrow the possible error types to a few categories, making it easier for the framework to intercept them before runtime.

Schema contracts are first-class citizens in declarative pipelines: the column names, types, and nullability of each table are explicitly declared, and the framework can validate upstream-downstream compatibility before runtime. For agents, this is equivalent to a machine-readable interface specification—there is a basis when generating code, and static checks are possible after generation. Data quality rules (such as non-null, uniqueness, value ranges, freshness thresholds) are likewise attached to datasets in declarative form, and the framework can intercept non-conforming data before writing, rather than discovering problems only when downstream reports fail. Together, these constraints form an environment that "fails fast": agent errors are caught at compile time or pre-check time, rather than silently propagating through the production chain. The more explicit the constraints, the less room the agent has to improvise, but the higher the success rate and verifiability of each generation—this is the engineering meaning of "constraints as guardrails."

The Compound Interest of Composability

Declarative pipelines are split into units by table or dataset, each with clear input-output contracts. Agents can independently generate, test, and replace any one of these units without having to understand all the details of the entire pipeline. This composability turns "agents maintaining large-scale pipelines" from impossible into engineering-manageable—when something goes wrong, you locate one unit, and when fixing it, you only change one unit.

What the 70% Figure Means

More than seventy percent of production pipelines written by agents is a figure worth breaking down.

On one hand, it shows that agents have crossed the usability threshold for tasks like data pipelines that are relatively structured and have clear feedback signals. If a pipeline is written incorrectly, tests fail, data is empty, and downstream alerts fire—the feedback loop is short and objective, which is exactly the environment agents excel in.

On the other hand, this proportion itself does not constitute a quality guarantee. A high share of agent-written pipelines does not mean these pipelines carry an equivalent share of critical business load. The original article does not provide a distribution weighted by importance, which is something readers need to be wary of—a reporting pipeline batch-generated by agents and a core transaction reconciliation pipeline handwritten by a senior engineer carry completely different weight.

What is truly informative is the direction of the trend: when agents become the primary authors of pipelines, the evolution direction of the framework is no longer determined solely by the usage habits of human developers, but must consider the distribution characteristics of agent-generated code. Declarative-first is the first clear signal of this shift.

Practical Implications for Data Teams

For teams evaluating pipeline technology stacks, the original article's argument points to several actionable directions.

Front-load constraints. Rather than letting agents freely generate procedural code and then having humans review it, use declarative frameworks to fix dependencies, schemas, and quality rules, and let agents work only at the goal description layer. The review target shifts from "hundreds of lines of code" to "a structured declaration," allowing human attention to focus on what truly requires judgment.

Measure framework selection by verifiability. To assess whether a pipeline framework is "agent-friendly," you can ask a concrete question: can the agent's output be statically verified without running the full pipeline? Declarative structures make this easier to achieve, which is the underlying reason they are favored in agent scenarios.

Preserve entry points for human judgment. Agents are good at generating and patching, but not at judging "should this pipeline exist" or "is this metric definition reasonable." Declarative frameworks write business semantics explicitly into declarations, which actually makes it easier for humans to intervene at the correct level of abstraction rather than getting stuck in implementation details during code review.

Conclusion

"Built for agents" is not a marketing slogan but a concrete engineering constraint: the framework needs to assume that the primary author is an agent and reallocate the responsibilities of generation and derivation accordingly. Spark Declarative Pipelines offers an answer—fold execution details into the framework, leave goal descriptions to agents, and trade constraints for verifiability. The 70% figure may continue to rise, or it may hit a ceiling when it reaches critical workloads, but the direction is clear: the user profile of data infrastructure is changing, and framework design must keep up.

One-line Summary

More than 70% of production Spark Declarative Pipelines are written by AI agents, and framework design must shift to being built for agents.

Paragraph Summary

The article notes that over 70% of production Spark Declarative Pipelines (SDP) are now written by AI agents, with the primary users of pipeline frameworks shifting from human engineers to agents. Traditional imperative APIs are human-friendly, but each step is a decision point the agent must correctly generate, and errors accumulate with pipeline length; declarative pipelines delegate execution derivation to the framework and leave goal description to the agent, thereby compressing the error surface and reducing verification cost. Explicit constraints (dependencies, schemas, data quality rules) act as guardrails for agents, and table-level composability makes automated maintenance of large-scale pipelines manageable. The article also cautions that the 70% share does not equal the share of critical business load, but the trend is clear: framework design needs to reallocate generation and derivation responsibilities for agent authors.