TRACER: Teaching User Simulators Behavioral Consistency, Not Surface Imitation

Why User Simulation Is a Critical Foundation for Interactive AI

Faithful user simulation is a long-underappreciated foundational capability in building, evaluating, and optimizing interactive AI systems. To train and test conversational agents at scale, we need simulators that can interact with systems in place of real users. The problem is that a simulator capable of generating plausible single-turn responses doesn't necessarily reproduce how user intent evolves — or how interactions ultimately resolve — in real conversations.

In other words, sounding like a real person and making decisions like a real person are two very different things. A simulator might produce natural, fluent responses turn by turn, yet the intent trajectory, conversion outcome, and purchase path across the full conversation can diverge drastically from real user behavior. This gap — "surface-style alignment" without "behavioral consistency" — is the core problem that TRACER, proposed in a new arXiv paper, aims to solve.

TRACER's Design: A Two-Stage Training Approach

From Supervised Fine-Tuning to Multi-Turn Reinforcement Learning

TRACER is a multi-turn user simulator that explicitly models the evolving intent of users and learns to align simulated behavior with real interaction trajectories. Training proceeds in two stages.

The first stage applies supervised fine-tuning (SFT) on real user conversations, giving the model a foundation in dialogue patterns and language style. The second stage introduces multi-turn reinforcement learning (RL) — the key differentiator that sets TRACER apart from standard dialogue models. Pure supervised learning can only teach the model to imitate local plausibility at each step; it cannot guarantee global consistency across the full conversation trajectory.

Supervised Fine-Tuning (SFT) trains a model directly on labeled real-world data, teaching it "what to output given a specific input." This approach naturally captures local language patterns, but its optimization objective is to minimize per-step prediction error — it doesn't care about the global direction of the conversation. Reinforcement Learning (RL) works differently: the model receives reward signals through interaction with an environment and optimizes for cumulative return rather than per-step accuracy. In a dialogue context, this means the model can learn multi-turn strategic behavior in service of an ultimate goal (such as closing a sale). The logic behind combining both stages is to first use SFT to build the "foundation" of language ability, then use RL to sculpt global behavioral strategy on top of it — avoiding the training instability that comes from RL exploring from scratch.

Addressing Two Core Challenges in Long Conversations

Applying reinforcement learning to long conversations faces two classic challenges: reward sparsity and credit assignment. The former refers to the fact that the final outcome (e.g., whether a sale occurs) isn't known until the conversation ends, leaving the intermediate steps without feedback. The latter refers to the difficulty of determining which turns actually contributed to the outcome once it appears.

TRACER addresses these through a hierarchical reward design that provides signals at both the outcome level and the trajectory level, combined with deviation-aware advantage modulation. This mechanism ensures the model attends not only to whether the final goal is achieved, but also to whether the full interaction path stays close to real trajectories — alleviating both reward sparsity and credit assignment.

Reward sparsity and credit assignment are classic challenges in sequential decision-making, and they're especially pronounced in dialogue. In an e-commerce customer service setting, for instance, a conversation might span more than ten turns, with whether the user ultimately places an order being the only clear signal — and the contribution of each intermediate turn being completely opaque. The Credit Assignment Problem was first articulated by Minsky in 1961: when an overall outcome occurs, how do you fairly attribute credit or blame to specific decisions in the sequence? Deviation-aware advantage modulation is one of TRACER's answers: by measuring how far a simulated trajectory deviates from the real reference trajectory, it applies differentiated gradient update weights to different time steps, giving the model a stronger corrective signal at critical moments where it "goes off track" — rather than treating all time steps equally.

Experimental Results: Near-Indistinguishable from Real Users

The paper evaluates TRACER on real customer service conversations organized into reference cohorts. TRACER-7B outperforms the strongest baseline by 11.4 points on the conversion F1 metric, while achieving the lowest scores on both group-level conversion rate error and semantic trajectory distance — and generalizing to out-of-distribution scenarios.

Even more compelling is the human Turing test: participants' accuracy in identifying TRACER-generated conversations approached random chance, indicating that the generated dialogues are natural enough to fool human evaluators. This result directly validates the simulator's practical value — if humans can't reliably tell the difference, then using it to train and evaluate AI systems has a credible foundation.

Dynamic Marketing Benchmark: A Counterintuitive Finding

Building on the TRACER simulator, the authors further propose the Dynamic Marketing Benchmark. This benchmark uses simulated interactions to jointly evaluate LLMs across two dimensions: "persuasive effectiveness" and "response quality."

It surfaces a strikingly counterintuitive finding: higher response quality does not necessarily correspond to higher conversion rates. A polished, logically coherent, linguistically elegant AI isn't necessarily better at closing deals. Persuasion is an art of timing, pacing, and understanding user psychology — not just saying the right things beautifully. This finding has direct implications for LLM applications in marketing, customer service, and other commercial contexts: you can't evaluate an AI's business value solely by looking at single-turn response quality scores.

This finding has academic resonance with a classic distinction in persuasion research: informativeness and persuasiveness are two independent dimensions. Response quality is typically measured by coherence, relevance, and grammatical fluency — metrics that are easy to quantify in single-turn evaluations and have therefore become standard on LLM leaderboards. But conversion rate reflects users' behavioral decisions after an entire conversation, shaped by factors like timing, objection handling, and emotional resonance — factors only loosely connected to how polished any individual sentence sounds. The value of the Dynamic Marketing Benchmark lies in bringing "persuasive effectiveness" into the evaluation framework, enabling benchmarks to capture signals that actually matter in commercial settings. The implication for LLM developers: optimization paths guided by BLEU, ROUGE, or human quality ratings may be misaligned with actual objectives in business dialogue scenarios.

Implications for the Industry

TRACER's value extends beyond being a specific model — it proposes a new paradigm for evaluating interactive AI. As more AI products are expected to complete tasks across multi-turn conversations, how to test their real-world performance at low cost and at scale has become an unavoidable bottleneck. Traditional single-turn evaluations or human testing are either distorted or prohibitively expensive.

A simulator that faithfully replicates real user behavior makes "using AI to test AI" genuinely viable. Meanwhile, the "quality-conversion decoupling" phenomenon it reveals serves as a reminder for practitioners to reconsider the objective functions they use to optimize their AI systems. For conversation-intensive domains like customer service, marketing, and sales, this kind of research offers evaluation tools that are far more aligned with what actually matters in business.