A/A Testing
A method used in causal inference to validate the stability of model estimates. It artificially creates a zero-treatment-effect scenario by randomly assigning a placebo treatment group using pre-treatment data, then examines the distribution of estimation error when the true effect is known to be zero. It is widely used to assess the reliability of different causal models.
Timeline (last 90 days)
A/A testing is used to measure the noise level of the test framework itself, and is a prerequisite for A/B testing.
An A/A test (placebo test) creates a scenario in which the true effect must be zero by using only pre-treatment data, pretending that treatment began earlier, and randomly selecting treatment groups, so that any non-zero effect reported by the model is treated as estimation error.
Claude Code identified that the pre-treatment trends of the treatment and control groups were not parallel, but did not further test a more accurate model and completely skipped the A/A testing step.
In the placebo test, 90% of the difference-in-differences estimates fell between -14% and +10%, with an RMSE of 7.6 percentage points.
The author repeated the placebo test for each model 500 times with random sampling in order to characterize the empirical distribution of errors and calculate RMSE.
When the true effect cannot be directly verified, a round of A/A testing can be run on candidate causal models first, and then the model to trust can be determined based on estimation error.
All Facts (6)
A/A testing is used to measure the noise level of the test framework itself, and is a prerequisite for A/B testing.
50%UnverifiedAn A/A test (placebo test) creates a scenario in which the true effect must be zero by using only pre-treatment data, pretending that treatment began earlier, and randomly selecting treatment groups, so that any non-zero effect reported by the model is treated as estimation error.
50%UnverifiedClaude Code identified that the pre-treatment trends of the treatment and control groups were not parallel, but did not further test a more accurate model and completely skipped the A/A testing step.
50%UnverifiedIn the placebo test, 90% of the difference-in-differences estimates fell between -14% and +10%, with an RMSE of 7.6 percentage points.
50%UnverifiedThe author repeated the placebo test for each model 500 times with random sampling in order to characterize the empirical distribution of errors and calculate RMSE.
50%UnverifiedWhen the true effect cannot be directly verified, a round of A/A testing can be run on candidate causal models first, and then the model to trust can be determined based on estimation error.
50%