Schema-Free Enterprise Data Synthesis: Breaking Down the STS Simulation Paradigm

The Enterprise AI Data Dilemma

Tool-calling agents have become a core component of enterprise AI — but a persistent challenge remains when it comes to training and evaluating these agents at scale: enterprise systems, data, and database schemas are often tightly locked down by trade secrets and legal compliance requirements. In other words, obtaining real enterprise data for model training is difficult, and large-scale agent evaluation in live environments is even harder.

This arXiv paper from the SAP team (arXiv:2610.10549v1) targets exactly this pain point, proposing a new data synthesis paradigm called Synthesis Through Simulation (STS). It sets out to answer a critical question: without access to real database schemas, how can you generate enterprise-grade data that is both structurally valid and distributionally faithful?

The Limitations of Existing Approaches

To appreciate what STS offers, it helps to first understand the shortcomings of current mainstream solutions. The paper identifies two common paths for addressing data scarcity — each with fundamental weaknesses.

Tabular Data Synthesis: Constrained by Structural Validity

The first path is tabular data synthesis. Statistical synthesizers can approximate the distributional characteristics of real data, but their effectiveness is fundamentally constrained by two factors: structural validity and schema availability. Making matters worse, many statistical synthesizers require seed data as input — which is precisely what's hardest to obtain in enterprise settings. Of the ten environments tested in the paper, seven could not use statistical synthesizers at all due to this hard dependency on seed data.

Tabular data synthesis is a classic direction in data augmentation, with common approaches including TVAE (based on variational autoencoders), CTGAN (based on generative adversarial networks), and more recently diffusion-model-based methods. What all these approaches share is a prerequisite: a batch of real data (seed data) must first be available to learn from. In public dataset settings (e.g., medical or financial academic datasets), this pipeline works well. But enterprise databases often involve multi-table relationships, foreign key constraints, and cross-field business rules. Even if a synthesizer learns the marginal distribution of individual columns, recreating these complex constraints is extremely difficult — the generated data may be logically inconsistent and rejected by real business systems.

Procedure-Based Methods: Lacking Distributional Fidelity

The second path is procedure-based methods. These generate data using pre-written rules, which ensures validity, but the weakness is the exact opposite: they lack distributional fidelity and typically require per-domain authoring, making them difficult to scale.

The two approaches present a clear dilemma: one preserves distribution but sacrifices validity; the other preserves validity but sacrifices distribution.

The Core Idea Behind STS: Let the Environment Define Validity

The breakthrough in STS lies in its schema-free design philosophy. Rather than relying on direct access to database schemas, STS has an LLM agent operate within a simulated enterprise environment, executing actions by calling policy-enforcing APIs to generate data.

There's an elegant insight here: because data is generated through the very same environment that defines what is valid, STS guarantees structural validity by construction. More importantly, it decouples the enforcement of validity from distributional modeling, allowing the two concerns to be handled independently — a direct solution to the dilemma described above.

Think of it this way: traditional methods either define rules first and then fill in data, or learn a distribution first and then patch for validity. STS instead lets the agent interact with the system like a real user, and the system's own APIs and policies automatically reject any invalid operations — validity is therefore naturally guaranteed.

The Generalist Populator: A Domain-Agnostic Filling Agent

With structural validity addressed by design, the remaining challenges are distributional fidelity and synthesis scalability. To tackle these, STS introduces the Generalist Populator (GP) — a domain-agnostic agent.

The experimental results reported in the paper are striking:

• The GP achieved an average marginal fidelity of 0.88 across all ten environments;
• It achieved a 100% constraint satisfaction rate;
• Critically, this was accomplished without any access to the database schema.

By comparison, statistical synthesizers could not be applied to seven of the ten environments due to seed data requirements. Even schema-privileged agents — those with full access to the schema — had up to 82% of trajectories fail in tightly coupled airline industry workflows, due to brittle task composition.

This comparison leads to a counterintuitive conclusion: agents with more information (schema access) do not necessarily perform better. In complex, tightly coupled business workflows, directly manipulating the schema is actually more prone to failure from task orchestration errors, while interacting via a simulated environment and APIs proves more robust.

"Marginal fidelity" is a standard metric for evaluating synthetic data quality, typically referring to how closely the univariate statistical distribution of each field in the synthetic data (e.g., mean, variance, category frequency) matches real data, often quantified using Wasserstein distance or KL divergence. An average marginal fidelity of 0.88 means the synthetic data closely matches the reference real data on most fields' distributions — though this metric cannot capture deviations in joint distributions across multiple variables. This is why the paper also emphasizes the "constraint satisfaction rate" as a complementary metric; together, the two provide a complete picture of synthetic data quality. Additionally, in the context of evaluating tool-calling agents, a "trajectory" refers to the complete sequence of actions an agent executes to complete a task — including API calls, parameter selections, and intermediate states — and is the fundamental unit for evaluating an agent's planning capability.

Open-Source Value and Industry Significance

Notably, the SAP team has open-sourced the entire framework, all ten simulation environments, and the generated datasets, with code hosted on GitHub (https://github.com/SAP/synthesis-through-simulation). For enterprise AI practitioners, this means they can train and evaluate tool-calling agents directly in these simulated environments without touching sensitive real enterprise data.

From a broader perspective, STS represents a shift in thinking about data synthesis: from statically imitating data to dynamically generating data. When data is produced through genuine interactions between an agent and a system, it is not only structurally valid — it also naturally carries business logic and operational context. This has real implications for training the next generation of enterprise-grade agents, especially in domains like finance, aviation, and supply chain where data compliance requirements are especially strict.

Of course, the paper also implies some boundaries yet to be verified: the cost of constructing simulation environments, how the GP performs under more extreme distributions, and whether the paradigm can generalize beyond the ten tested environments — all of these remain to be validated through further research and community practice. But as a methodological innovation, STS offers a compelling new path for scaling enterprise data synthesis within compliance constraints.