How to Choose a Causal Inference Model? Use A/A Placebo Tests to Avoid Estimation Bias

In machine learning, model selection is a well-solved problem — cross-validation, holdout sets, and evaluation metrics are all standard practice. But in causal inference, model selection is frequently skipped. The reason is straightforward: you can never observe the "true effect." How much did the treatment actually lift the outcome? There's no parallel universe to check. A data science practitioner shared a clever validation framework on Reddit, using simulated experiments to reveal striking accuracy differences between causal models.

Creating "Known Ground Truth" with Simulated Data

To evaluate whether a causal model is accurate, you need to know the right answer first. The author's approach was to generate the data themselves: they simulated weekly booking volumes across 40 regions over two years, with 5 regions receiving a "treatment" at a specific point in time (the red vertical line in the chart). Because the data was synthetically generated, they knew the ground truth — the treatment increased bookings in those 5 regions by exactly 10%.

With 5 treated units and 35 control units, the two most natural candidates were Synthetic Control and Difference-in-Differences (DiD) — plus other methods. The key question: when you can't directly verify the answer, which one should you trust?

Synthetic Control Method, introduced by Abadie, Diamond, and Hainmueller in 2010, works by constructing a weighted combination of control units that approximates the counterfactual — what the treated units would have looked like had the treatment never occurred. Weights are determined by fitting pre-treatment data, with the goal of making the synthetic control track the treated group as closely as possible before treatment. Unlike DiD, it does not require the parallel trends assumption and constructs a separate match for each treated unit. This makes it particularly well-suited when there are few treated units (just 5 regions here) and many control units. Its limitation is that fit quality degrades when there are many treated units or when the donor pool is insufficient.

Claude Code's Blunder: 24% vs. 10%

The author handed the dataset entirely to Claude Code and let it run freely — with cautionary results. Claude built its entire analysis around Difference-in-Differences and concluded the treatment produced a 24% increase — more than double the true effect of 10%.

To Claude's credit, it did flag a warning sign: the pre-treatment trends for the treated and control groups weren't parallel (a violation of the parallel trends assumption, which is the core requirement for DiD to be valid). But it stopped there, never testing whether a more accurate model existed. More critically, Claude skipped an essential step entirely: the A/A test (also called a placebo test).

This perfectly illustrates the author's point: doing model selection well is one of the things that separates senior data scientists from junior practitioners. A tool can produce a number — but deciding whether that number is trustworthy requires methodological rigor.

The Parallel Trends Assumption is the foundational requirement for DiD: in the absence of treatment, the treated and control groups must have followed the same trend over time. Once this assumption is violated — for example, if the treated group was already growing at a different rate before treatment — DiD will incorrectly attribute pre-existing trend differences to the treatment effect, introducing systematic bias. This is why, after identifying that "trends aren't parallel," the correct next step is to switch to a method less sensitive to that assumption (like Synthetic Control), not to continue with DiD and report the result.

A/A Testing: Manufacturing "Zero Truth" Where There Is None

The A/A test is elegantly simple. You take only the data from before the real treatment occurred, pretend the treatment started three months earlier than it did, and randomly pick 5 regions as fake "treated" units. Since nothing actually happened in this time window, the true effect is necessarily zero.

The logic then becomes clear: any nonzero effect the model reports is pure estimation error. The author repeated this process 500 times with different random samples, fully characterizing the distribution of errors. This gives you a ruler — a way to measure how far the model drifts in a scenario where the answer should be flat zero.

The name "A/A test" comes from analogy to A/B testing: while A/B testing compares two different versions, A/A testing deliberately makes both groups identical — in the causal inference context, this means artificially constructing a "zero treatment effect" scenario. In statistics, this is also known as a placebo test, widely used to verify that an estimator behaves correctly under the null hypothesis. By repeating random sampling (500 times here), you get an empirical distribution of errors, from which you can compute RMSE (root mean squared error) as an objective basis for comparing model stability. The elegance of this approach is that it requires no assumptions about the true effect — only that you have a sufficiently long pre-treatment history.

Results: Synthetic Control Wins Clearly

The simulation results are compelling:

• Difference-in-Differences: 90% of placebo estimates fell between -14% and +10%, with an RMSE (typical distance of estimates from the true zero) of 7.6 percentage points.
• Synthetic Control (same placebo sampling): 90% of estimates narrowed to -7% to +9%, with RMSE dropping to 4.9 percentage points — more than one-third lower than DiD.

The author also highlights an easily overlooked decision: the choice of outcome variable itself matters. He applied Synthetic Control to "bookings per thousand population" rather than raw bookings. Putting all regions on the same scale allows the model to track the treated group more tightly in the pre-treatment period, and RMSE dropped further to 4.4 percentage points.

Back to the Real Treatment Period: Rankings Confirmed

The most satisfying result: the model rankings from the placebo test were validated on the real treatment period.

• Synthetic Control (raw bookings): estimated 7%
• Synthetic Control (per-thousand population): estimated 9.6%
• Difference-in-Differences: estimated 24%

The first two are clearly closer to the true 10%, while DiD's bias is unmistakable.

The author maintains appropriate restraint: based on a single simulated dataset, he won't over-interpret the small gap between the two Synthetic Control variants, nor claim that Synthetic Control is "always better." What's truly transferable to real work is the workflow itself — in practice you'll never see the ground truth, but you can almost always run an A/A test on your candidate models before deciding which one to trust.

Key Takeaways for Causal Inference Practitioners

This case offers three layers of insight. First, it's a reminder not to blindly trust automated AI analysis: Claude Code could identify a violated assumption, but it won't proactively do model selection and validation for you. Second, placebo testing provides a practical, low-cost, reproducible way to quantify model error in a "no ground truth" environment. Third, it highlights an underappreciated decision — the choice of outcome variable (raw vs. normalized) also affects estimation accuracy.

The author has made the code (Colab) and an explanatory video freely available. For any team doing experiment evaluation or policy effect estimation, incorporating A/A testing into your standard workflow may matter more than agonizing over which model to pick.