Synthetic Query Diversity: Useful, or an Illusion?

E-commerce search's cold-start problem usually comes down to labeled data. Using LLMs to synthesize queries for training semantic parsers looks like a shortcut — but only if the synthetic diversity is "useful" rather than "hallucinated variety." A study focused on the UK automotive e-commerce scenario breaks this question down and quantifies it.

Experimental Design: How Much Synthetic Data

The study constructed 118,098 synthetic query-intent pairs, generated by 6 LLMs across 9 persona-conditioned prompt families, covering a 47-field UK automotive schema. The control groups included 1,214 human-annotated queries and 15,304 real production queries. This data scale is sufficient to separate and test the two variables of "prompt diversity" and "generation faithfulness."

The Real Gains from Persona Conditioning

The most direct conclusion: IPIP persona combinations improved the semantic parser's F1 in all four matched comparisons (Holm-corrected p<.001), and all three persona combinations had positive main effects across the full 2³ grid. This means persona conditioning doesn't just make sentences read more like human speech — it genuinely pushes the generated data toward structured intents.

But the three combinations did not act symmetrically. Grammar conditioning was the axis that pushed the generated distribution toward real traffic, while also bringing lexical-level diversity. In other words, the genuinely useful diversity comes from structural and grammatical constraints, not merely from changes in persona tone.

IPIP (International Personality Item Pool) is a public-domain item bank used to measure the Big Five personality traits. So-called "persona-conditioned prompts" refer to injecting specific personality descriptions into the prompt (e.g., "You are a rigorous, organized person") to guide the LLM to generate queries in a tone and style consistent with that persona. In this study, the three persona combinations correspond to different Big Five trait combinations and were orthogonally designed against grammar conditioning (e.g., forcing queries to contain specific syntactic structures or field constraints), enabling the main effects and interaction effects of the two to be separated across the 2³ grid.

Predicting Parser Performance with a Label-Free Metric

The study proposed the aligned sKL metric: fitting von Mises-Fisher distributions separately to synthetic queries and human-annotated queries, and taking the divergence between them. It requires no labels yet can serve as a proxy metric for parser F1 — when compared against human queries written for simulated shopper scenarios, the correlation coefficient reached ρ=-0.917, with lower values indicating closer alignment.

The value of this metric is that it lets teams predict the quality trajectory of synthetic data before training the parser, rather than discovering distribution shift only after completing a round of training.

A von Mises-Fisher (vMF) distribution is a probability distribution on a sphere, analogous to a Gaussian distribution for directional data. After normalizing query embedding vectors onto the unit sphere, the vMF distribution can characterize the concentration and dispersion of these directional vectors. Aligned sKL (symmetric Kullback-Leibler divergence) is the symmetric divergence between two vMF distributions, used to measure the distributional difference between synthetic and human queries in embedding space. Since divergence computation does not depend on intent labels, it can serve as a label-free proxy metric for data quality before parser training.

What's Hiding in the Queries All Parsers Missed

The study used an LLM panel to triage 927 queries that all parsers missed, with blind author review as validation, and ultimately judged 37% of them to be hallucinations. The main form of hallucination was "lost constraints" — generated queries dropped key restrictive conditions from the original intent, looking superficially rich in sentence structure while the actual semantics had drifted.

Broken down by family, the four prompt families with grammar conditions had the highest hallucination proportions, reaching 38% to 42%; the family using only IPIP personas had the lowest at 28%. This creates a tension with the earlier conclusion that "grammar conditioning brings diversity": while grammatical constraints bring the distribution closer to real traffic, they also amplify the risk of lost constraints.

An "LLM panel" refers to having multiple LLMs independently judge the same query (e.g., determining whether it is a hallucination), then aggregating conclusions through voting or consensus mechanisms. Its purpose is to reduce the bias of any single model's judgment and improve annotation reliability. In this study, the panel's triage results were further validated by blind author review to control for false positives.

Implications for Cold-Start Search Deployment

The practical takeaway from this study is that synthetic query diversity needs to be managed by splitting it into "useful" and "hallucinated" parts. Persona conditioning is worth using — it reliably improves parser F1; grammar conditioning is more critical for distribution alignment but must be paired with hallucination detection, especially checks for lost constraints. Label-free proxy metrics like aligned sKL can filter out generation batches that clearly deviate from the real distribution before training.

For cold-start e-commerce search teams, this means the focus of the synthetic data pipeline is not "generate more" but "verify after generation." Diversity itself is not the goal — diversity that is faithful to structured intents is.