Simulated Control Evaluations: A New Paradigm for Predicting AI Safety with Language Models

The Bottleneck of Control Evaluations: The Gap Between the Lab and Real Deployment

Control Evaluations are designed to measure whether safety safeguards can successfully prevent harm even when an AI model intentionally causes adverse outcomes. These evaluations typically rely on purpose-built control environments and then extrapolate the results to real deployment scenarios. However, safety effectiveness in deployment environments often depends on task characteristics and environmental variables that control environments fail to capture—meaning that safety conclusions drawn in the lab may not hold in the real world.

According to the research team's experiments on LinuxArena, the traditional control setup BashArena has a root mean square error (RMSE) of 10.1 percentage points (under a 5% audit budget and five monitor conditions) when predicting deployment safety. This level of error is clearly too coarse for safety decisions involving high-risk AI deployments.

Control Evaluations are a paradigm in AI safety evaluation whose core assumption is that a model may have malicious intent or may be exploited by an attacker. During evaluation, a red team (simulating an attacker) attempts to induce the model to perform dangerous operations (such as deleting files or leaking data), while a blue team (the defender) deploys safeguards such as monitors and audit budgets. The goal is to measure whether the safeguards can successfully prevent harmful behavior under a given attack strength. Such evaluations typically focus on "worst-case" safety rather than the model's capability ceiling.

Audit Budget refers to the proportion of suspicious operations that the defender can manually review. For example, a 5% audit budget means the system automatically flags 5% of tool calls for human review, with the rest relying on automated monitoring. The lower the audit budget, the harder defense becomes, so evaluations are often conducted under low-budget conditions to test the limits of protection.

Root Mean Square Error (RMSE) is a common metric for measuring the deviation between predicted and true values; when expressed in percentage points, it indicates the average magnitude of error in predicted safety probabilities. RMSE is more sensitive to large errors, making it suitable for evaluating the reliability of safety predictions.

Simulated Control Evaluations (SCEs): Replacing Real Environments with Language Models

To solve this extrapolation problem, researchers proposed Simulated Control Evaluations (SCEs): using language models to simulate tool responses in place of a real execution environment. This way, control evaluations are no longer constrained to a specific control setup and can instead be flexibly constructed for concrete deployment scenarios.

The core idea of SCEs is not complicated: since real environments are costly to execute and limited in coverage, why not let a language model play the role of the environment itself, generating tool call results and thereby completing the evaluation in simulation? The key question is whether this simulation is sufficiently faithful and whether it can accurately predict safety performance in real deployments.

Validation Method: LinuxArena as Ground Truth

The research team validated SCEs on LinuxArena because this environment can provide real post-execution results as a benchmark. They compared the prediction error of SCEs with that of the BashArena control setup, while also examining the natural variation among different control metrics within LinuxArena. This design allows the accuracy of simulated predictions to be rigorously quantified.

KL Divergence (Kullback-Leibler Divergence) measures the difference between two probability distributions: the smaller the value, the closer the distributions; a value of 0 means they are identical. In this article, KL divergence is used to quantify how similar the simulated environment and the real execution environment are in terms of behavioral distributions such as attack strategies, attacker reasoning, and monitor reasoning. A more than tenfold reduction in KL divergence means that the behavioral patterns produced by the simulated environment are highly consistent with those of the real environment, rather than merely having similar average metrics. This metric reflects simulation fidelity better than simply comparing success rates, because the effectiveness of safety evaluation depends on accurately characterizing attack diversity and monitor decision boundaries.

A Leap in Accuracy: 3.6 Percentage Points vs 10.1 Percentage Points

On independently sampled evaluation and deployment task sets, SCEs predict safety with a root mean square error of only 3.6 percentage points (also under a 5% audit budget and five monitor conditions), compared with 10.1 percentage points for BashArena. The former's error is less than 40% of the latter's—a significant improvement.

Even more noteworthy is SCEs' performance in distributional fidelity. The research shows that SCEs closely reproduce the distributions of attack strategies, attacker reasoning, and monitor reasoning, with KL divergence from real execution more than ten times lower than that of BashArena. KL divergence measures the difference between two probability distributions, and a more than tenfold reduction means the simulated environment is highly close to the real environment in behavioral patterns, not just numerically similar.

In addition, SCEs can explain 78% to 85% of the variance across environments in attack success rate, monitor score, and safety. This explanatory power indicates that simulated evaluation captures the main drivers of environmental differences rather than merely fitting surface-level data.

Proportion of Variance Explained (R²) indicates the proportion of variation in the target variable that the model's predictions can explain. In this article, SCEs explain 78% to 85% of the variance across environments, meaning that most of the differences among deployment environments in attack success rate, monitor score, and safety can be explained by the environmental features captured by SCEs, rather than by random fluctuations or unobserved factors. High explanatory power shows that simulated evaluation is not only accurate in prediction but can also identify the key drivers affecting safety, providing attributable evidence for deployment decisions.

Sample Efficiency and Policy Transfer: The Extra Benefits of Simulation

SCEs' accuracy is also reflected in sample efficiency. The research shows that SCEs use fewer than 5% of the task-pair samples in the deployment task set while predicting safety with lower error. For control evaluations that require large numbers of task pairs, this efficiency advantage means lower evaluation costs and faster iteration cycles.

Another key finding is policy transfer: attack strategies discovered through simulation also improve monitor performance on held-out LinuxArena attacks, outperforming BashArena and roughly matching the improvements brought by real execution strategies. This shows that simulation is not only used for prediction but can also generate valuable training signals that feed back into actual defensive capability.

Policy Transfer refers to whether attack strategies discovered in a simulated environment (such as specific prompt injection sequences or tool call combinations) are equally effective in the real environment, and whether monitors trained on simulated attacks can perform well on real attacks. This article finds that after attack strategies discovered in simulation are used to train monitors, their improvement on held-out real attacks matches that of real execution strategies, indicating that simulation can not only be used for prediction but can also generate high-quality adversarial examples that feed back into defense systems. This characteristic gives SCEs both evaluation and training value.

Implications for AI Safety Evaluation

This research sends a clear signal: deployment-specific simulated control evaluations may be more predictive than control evaluations executed in independent control setups. Traditional methods assume the representativeness of the control setup, whereas SCEs model the deployment scenario directly, bypassing the distortion introduced by the extrapolation step.

Of course, the effectiveness of SCEs depends on the fidelity of the simulated language model's responses to the environment. The KL divergence data indicate that the current approach is already quite reliable, but the differing complexity of various deployment scenarios still requires further investigation. For the AI safety field, SCEs offer a scalable, low-cost, and high-fidelity evaluation path that is worth validating and promoting in more environments.

One-line Summary

Simulated control evaluations replace real environments with language models, predicting safety with an error of only 3.6 percentage points—far better than the 10.1 percentage points of traditional methods.

Paragraph Summary

The article explores the distortion that occurs when AI control evaluations are extrapolated from the lab to real deployment, noting that the traditional control setup BashArena has a root mean square error of up to 10.1 percentage points in predicting safety. To address this, researchers proposed Simulated Control Evaluations (SCEs), which use language models to simulate tool responses and flexibly construct evaluations for specific deployment scenarios. Validation on LinuxArena shows that SCEs reduce prediction error to 3.6 percentage points, achieve more than ten times lower KL divergence from real execution, explain 78% to 85% of the variance across environments, and are highly sample-efficient. Attack strategies discovered in simulation can also transfer to real environments to improve monitor performance. SCEs provide a scalable, low-cost, and high-fidelity path for AI safety evaluation.