How Much Synthetic Data Do Activation Probes Need? Coverage Calls the Shots, Not Difficulty

Background: The Mystery of How Much Data Probe Training Needs

Large language models deployed in production often require "monitoring probes" (activation probes) to observe the model's internal states in real time — detecting whether a conversation falls into a high-stakes scenario, whether the model has generated a reply harmful to the user, or whether it has deviated from user instructions. These probes are typically trained on synthetic conversation data, but a long-standing open question remains: how many synthetic samples does it actually take for a probe to reach usable performance?

A newly released arXiv preprint (arXiv:2610.10594v1) systematically answers this question. The research team tracked probe learning curves across a range of 10 to 590 synthetic samples, covering three monitoring concepts, fourteen held-out evaluation distributions, and four different probe models — while also varying the LLM used to generate data and the level of detail in the prompts.

The conclusion points to a counterintuitive core finding: what determines data requirements is not the inherent "difficulty" of a given concept, but rather coverage.

Activation probes are typically lightweight linear classifiers attached to the hidden states of a specific layer in a large language model, mapping high-dimensional activation vectors to binary or multi-class labels. Unlike methods that analyze model output text directly, activation probes read the model's internal representations at inference time, making them extremely low-latency and well-suited for online monitoring. The motivation for using synthetic conversation data is that real-world high-stakes or harmful dialogue samples are often scarce, expensive to annotate, and carry privacy and compliance risks. Researchers therefore typically use an LLM to automatically generate large numbers of (concept, dialogue) sample pairs based on given concept descriptions. The core assumption underlying this approach — that the semantic distribution of synthetic data sufficiently covers the real-world target concept — is one of the central questions this paper investigates.

Differences in Learning Curves Across Three Monitoring Concepts

The study focuses on three representative monitoring tasks:

• High-stakes situations
• Replies harmful to a person
• Replies that do not follow user instructions

Experiments show that on Gemma-3-27B-IT as the probe model, probes for "high-stakes" and "harmful replies" need only around 80 samples to approach their performance plateau (within a few percentage points). The "instruction-following" probe, however, requires several times as many samples.

Crucially, this ordering relationship is robust: it holds across three other smaller probe models, and is confirmed on real samples from the development set as well. In other words, the differences in data requirements across monitoring tasks are fundamental, not an artifact of any particular model or synthesis method.

Coverage vs. Difficulty: Findings from Variance Decomposition

To quantify "how deep a concept's data requirements go," the researchers introduced the half-gain size metric — defined as the number of samples needed on a fitted learning curve to achieve half of the total performance gain.

Through variance decomposition, the team uncovered a clear attribution structure:

• Concept and distribution account for 42%–45% of variance
• Generator LLM, probe model, and prompt detail level together contribute less than 10%

This means that which large model you use to generate data, and how detailed your prompts are, has surprisingly little impact on the final data requirements. What truly matters is the concept being monitored and the structural properties of the data distribution.

Going further, the paper argues that what determines half-gain size is coverage, not per-kind difficulty. Specifically, if each evaluation distribution is treated as a "kind" of data, the median number of within-kind samples needed for saturation is consistently in the range of 7–11, across all three concepts. The learning efficiency within a single data kind is nearly uniform.

The half-gain size definition comes from a parametric fit of learning curves. Researchers typically fit the relationship between probe performance and sample count as a logarithmic or power-law curve, with "half-gain size" corresponding to the sample count at which the cumulative performance gain (from baseline to asymptotic plateau) reaches 50%. The advantage of this metric is that it is independent of absolute performance levels and captures only "learning speed," enabling fair comparisons across different evaluation distributions and concepts. Variance decomposition uses a framework analogous to ANOVA, decomposing the total variation in half-gain size across different factors (concept, distribution, generation model, probe model, prompts). The fact that 42–45% of variance is explained by concept and distribution indicates that differences in data requirements are primarily determined by what is being monitored, not how the data is generated.

The Real Difference: Cross-Kind Transfer

Given that the learning curves within each data kind are so similar, where do the large differences between the three concepts come from?

The answer is cross-kind transfer — the degree to which capabilities learned from one kind of data generalize to other kinds within the same concept.

• Under the high-stakes concept, transfer is nearly complete: one kind of data essentially covers all the others
• Under harmful replies, transfer is moderate
• Under instruction-following, transfer is weakest: almost no kind of data covers another

This difference in transfer is the primary source of the performance gap between synthetic and real samples observed across the three concepts.

Implications for Data Generation Strategy: Breadth or Depth?

The prevailing advice in prior work is that under a fixed generation budget, one should prioritize breadth (more kinds of data) over depth (more samples per kind). This paper offers a more nuanced revision: the returns from breadth vary by concept.

• For instruction-following tasks, breadth is critical — because no single kind of data covers another, you must collect as many kinds as possible
• For high-stakes tasks, breadth is nearly redundant — since one kind covers the rest, spending budget on variety yields diminishing returns

This provides actionable decision-making guidance for real-world data collection: first assess the cross-kind transfer properties of the target concept, then decide whether to invest the budget in breadth or depth.

Open Resources and Broader Value

The research team has publicly released the evaluation suite, development set, and generated datasets to facilitate reproduction and extension by future work. For researchers and engineers working on LLM safety monitoring and model behavior probing, this work provides a clear framework: don't blindly pile on synthetic data volume. Instead, examine the transfer relationships across data kinds through the lens of coverage, and allocate your limited generation budget where it counts most.

Additional Context

The strength of cross-kind transfer intuitively correlates closely with the semantic cohesiveness of a concept. "High-stakes situations" is a relatively unified semantic category — whether it's a medical emergency or a domestic crisis, the core features that trigger a "high-stakes" judgment (urgency, physical safety threats) overlap substantially, so the feature vectors learned from one kind of training data generalize well to other subkinds. "Not following user instructions," by contrast, is a highly heterogeneous concept cluster: failure modes such as outright refusal, partial execution, format changes, and ignored constraints differ significantly in their activation patterns, making a probe trained on one failure mode nearly unable to recognize another. This semantic heterogeneity is the fundamental reason for cross-kind transfer failure, and it explains why instruction-following probes require far more samples than probes for other concepts.