In Praise of Observational Evidence: When Randomized Controlled Trials Can't Answer Everything

The Over-Mythologized Randomized Controlled Trial

In the world of evidence-based decision-making, the Randomized Controlled Trial (RCT) has been elevated to the apex of the evidence pyramid. Whether in medical research, policy evaluation, or data science, the prevailing assumption is that only conclusions validated through randomized experiments deserve to be called "reliable evidence." Observational evidence, by contrast, is routinely labeled as "second-rate," "riddled with bias," or "not worth trusting."

Yet this black-and-white view is drawing increasing pushback from researchers. An article titled In Praise of Observational Evidence sparked a lively discussion on Hacker News, prompting a fresh look at the irreplaceable value of observational data in scientific discovery. This piece explores why observational evidence deserves to be reconsidered.

What Is Observational Evidence?

Definition and Basic Forms

Observational evidence refers to data obtained when researchers observe, record, and analyze naturally occurring phenomena without actively intervening on the subjects being studied. Unlike RCTs — which involve deliberately assigning subjects to groups and applying interventions — observational research simply "records the world as it is."

Common observational study designs include cohort studies, case-control studies, and cross-sectional studies. They are widely used in epidemiology, economics, social sciences, and even astronomy.

• Cohort studies follow a group of people who share common characteristics over time, tracking health or behavioral changes. They are well suited for studying associations between exposures and long-term outcomes. A prime example is the UK Biobank project, which enrolled 500,000 participants for decades of follow-up.
• Case-control studies start from outcomes and work backward, comparing historical exposure differences between a diseased group and a healthy control group. They efficiently identify risk factors for rare diseases with relatively small sample sizes.
• Cross-sectional studies capture a population snapshot at a specific point in time, useful for describing disease burden and characteristic distributions — though their capacity for causal inference is relatively limited.

A Long-Underappreciated Truth

Many of humanity's greatest scientific breakthroughs came from observation, not experiment. Astronomy is built almost entirely on observation — no one can "randomly assign" stars to a control group. Darwin's theory of evolution, the theory of plate tectonics, and the established link between smoking and lung cancer all originated from careful observational accumulation. These examples remind us that scientific rigor is not synonymous with any single method — including randomized experiments.

The establishment of the causal link between smoking and lung cancer is itself a landmark for observational research. In the 1950s, Richard Doll and Austin Bradford Hill built an overwhelming body of epidemiological evidence through large-scale case-control and cohort studies. Hill subsequently proposed his famous "Hill's Criteria" — nine standards for causal judgment including strength of association, consistency, specificity, temporality, biological gradient, and plausibility. These criteria remain a core reference framework for causal argumentation in observational research to this day. Their significance lies precisely in providing a systematic methodology for evaluating causality in domains where randomization is impossible.

Background: The Historical Rise of RCTs

The modern form of the randomized controlled trial dates to 1948, when the British Medical Research Council conducted a clinical trial of streptomycin for tuberculosis. Designed by statistician Austin Bradford Hill, it was the first to introduce random group allocation and blinded assessment into medical experimentation, and is widely recognized as the founding work of modern evidence-based medicine. RCTs subsequently became the cornerstone of drug approval and clinical guideline development. The Evidence-Based Medicine movement of the 1990s further institutionalized this hierarchy, establishing a "pyramid of evidence" with systematic reviews and RCTs at the top. However, this framework was originally designed for clinical drug evaluation — applying it uncritically to policy, social science, and business decision-making is itself worth questioning.

Why Observational Evidence Has Been Undervalued

The Shadow of Confounding

The most common criticism of observational research is confounding variables. Because study subjects are not randomly assigned, observed correlations may not represent causal relationships — they could be driven by an unmeasured third factor. The classic example: ice cream sales and drowning rates are positively correlated, not because one causes the other, but because summer heat is the confounding variable.

Confounding has a particularly striking statistical manifestation: Simpson's Paradox — a trend observed in grouped data can completely reverse when the data are combined. The 1973 UC Berkeley admissions data is the canonical case: aggregate data showed a higher admission rate for male applicants, suggesting gender discrimination; but when stratified by department, female applicants actually had higher admission rates in most departments. The true confounder was departmental choice — women disproportionately applied to more competitive departments. This case vividly illustrates that confounding can not only create spurious correlations, it can completely invert the direction of a conclusion.

Simpson's Paradox is not merely a statistical curiosity — it has a rigorous mathematical foundation. When computing weighted averages across strata, differing sample size weights across layers can reverse the combined trend. From a linear algebra perspective, this reflects the directionality of vector addition; from a probability theory perspective, it involves the non-monotonic relationship between conditional and marginal probabilities. Judea Pearl reinterpreted Simpson's Paradox as a causal inference problem: whether to "control" for a variable depends on its role in the causal graph — it should be controlled if it is a confounder, but not if it is a mediator. The wrong control strategy itself creates the paradox. This insight elevates the paradox from a "counterintuitive statistical phenomenon" to a core warning in causal inference.

It is precisely because of traps like these that RCTs were elevated to their privileged status: randomization theoretically balances both known and unknown differences across groups, more cleanly isolating causal effects.

But the "Gold Standard" Has Its Own Costs

RCTs are far from perfect, and their limitations are often overlooked:

• Ethical constraints: We cannot randomly assign people to smoke for decades in order to study the harms of tobacco.
• High costs: Large RCTs can cost hundreds of millions of dollars and take years — many research questions simply cannot afford this.
• Limited external validity: To ensure clean internal causal inference, RCTs typically use strict inclusion/exclusion criteria, producing highly homogeneous study populations that may look nothing like real-world patients. Medical researchers call this the "clinical trial paradox" — the stricter the internal validity, the harder it is to generalize to the real world. In 2018, the U.S. FDA formally released a Real-World Evidence (RWE) framework, allowing evidence from observational sources such as electronic health records and insurance claims data to support drug approvals — a significant official reassessment of the status of observational evidence.
• Some questions simply cannot be randomized: Questions involving historical events, rare phenomena, or macro-level systems cannot be studied through randomization at all.

The FDA's RWE framework marks a major turning point in global drug regulation history. The 21st Century Cures Act of 2016 first required the FDA legislatively to develop guidance on RWE use, paving the way for the 2018 framework. The European Medicines Agency (EMA) has been pursuing similar policies in parallel. Core RWE data sources include electronic health records (EHRs), insurance claims data, patient registries, and wearable device data. Yet the application of RWE is not without controversy — uneven data quality, coding errors, and selective recording continue to limit its reliability. Establishing quality standards and review frameworks for RWE remains a frontier issue in regulatory science, and reflects the complex paradigm shift of observational evidence moving from "questioned" to "institutionally accepted."

In short, when RCTs are infeasible or prohibitively costly, observational evidence is often the only path available.

How Modern Methods Are Empowering Observational Evidence

The Rise of Causal Inference Tools

Over the past two decades, advances in statistics and computer science have substantially strengthened the credibility of observational data. The causal inference framework pioneered by Judea Pearl — including tools like Directed Acyclic Graphs (DAGs) and the backdoor criterion — gives researchers systematic ways to identify and control for confounding.

Judea Pearl is a Turing Award laureate and professor at UCLA, often called the "father of causal inference." In his book Causality (2000) and the popular work The Book of Why (2018), Pearl systematically proposed the "Ladder of Causation" — three levels of human cognition: association (seeing), intervention (doing), and counterfactual (imagining). Pearl argues that traditional statistics can only address questions at the associational level, while genuine causal inference requires structural causal models (SCMs) and DAGs as a formal language. DAGs not only visualize causal assumptions between variables, but also allow the backdoor criterion to precisely derive the minimal set of variables needed to control for confounding — transforming observational causal analysis from an "art" into an operational "science."

Beyond DAGs, quasi-experimental methods such as Propensity Score Matching, Instrumental Variables (IV), Regression Discontinuity (RD), and Difference-in-Differences (DiD) allow researchers to approximate causal conclusions without randomization. A classic IV application comes from Nobel laureate Joshua Angrist's study of the Vietnam draft lottery and earnings — lottery numbers served as a "naturally randomized" instrument, successfully stripping out the self-selection bias between military service and income. Regression Discontinuity exploits the near-identical characteristics of individuals on either side of a policy threshold for local causal estimation. Difference-in-Differences relies on the "parallel trends assumption," exemplified by Card and Krueger's landmark 1994 study on minimum wage and employment.

However, the credibility of these methods depends heavily on whether their underlying assumptions hold. DiD's parallel trends assumption requires that the treatment and control groups have identical pre-intervention time trends — an assumption that cannot be directly tested, only indirectly assessed by examining trends across multiple pre-treatment periods. Regression Discontinuity requires that there be no "precise manipulation" at the policy threshold — that is, individuals cannot precisely control which side of the threshold they fall on (exam-score-based thresholds, for instance, carry manipulation risk). IV requires that instruments satisfy the "exclusion restriction" — that the instrument affects the outcome only through the treatment variable — an assumption that is often difficult to falsify. Because each method carries stringent preconditions, sensitivity analysis and placebo tests have become standard accompaniments to evaluate credibility, reminding practitioners that quasi-experimental methods are not shortcuts around rigor, but rather rigorous arguments made under specific assumptions.

These methods, each with strict conditions for valid application, have become mainstream analytical tools in economics, public health, and related fields.

New Opportunities in the Age of Big Data

In the era of data science and AI, observational data has reached unprecedented scale and dimensionality. Electronic health records, user behavior logs, and sensor networks generate vast quantities of observational data, providing rich material for discovering patterns. While "big data is not the same as good data," combining large-scale observational data with rigorous causal inference methods can reveal long-tail phenomena and real-world effects that RCTs struggle to reach.

A notable development is the emergence of Causal Machine Learning — the fusion of large-scale observational data with machine learning. Traditional machine learning excels at finding predictive patterns in data, but prediction is not the same as understanding causation. Algorithms like the "Causal Forest" developed by Susan Athey (Stanford economics professor) and colleagues combine the non-parametric flexibility of random forests with causal inference frameworks to estimate Heterogeneous Treatment Effects — the differential impact of the same intervention on different subpopulations. This type of fine-grained insight is precisely what large-sample observational data can support, and represents the methodological foundation for the future of personalized medicine and precision policy.

A Rational View of Evidence: Complementary, Not Competing

Evidence as a Landscape, Not a Hierarchy

Framing observational evidence and experimental evidence as adversaries is itself a misconception. A more mature view holds that different types of evidence each have their own appropriate contexts and should be used in a complementary fashion. RCTs excel at verifying the causal effects of specific interventions under controlled conditions; observational research excels at discovering patterns, generating hypotheses, and validating whether experimental conclusions hold up in real-world, large-scale, and long-term settings.

The ideal scientific process often weaves both together: observation generates hypotheses, experiments test hypotheses, and then broader observation validates whether conclusions generalize. This cycle is also reflected in specific research designs — "nested studies" combine observational cohorts with randomized sub-experiments, balancing scale with causal rigor; "triangulation" strategies require that the same conclusion hold across multiple study designs, compensating for the limitations of any single method through methodological diversity.

Practical Implications for Practitioners

For data scientists, product analysts, and policymakers, this discussion has direct practical relevance. In many business settings, A/B testing (which is essentially an RCT) is not always feasible — whether due to insufficient sample sizes, ethical concerns, or prohibitive engineering costs. In these situations, the ability to perform causal analysis on observational data is often the difference between a "data mover" and a truly insightful analyst.

Specifically, practitioners need to develop three core competencies. First, causal graph thinking — before conducting any analysis, sketch out the hypothesized causal relationships between variables, identifying which are confounders, which are mediators, and which are colliders, to avoid mistaken variable control strategies. Second, methodological matching ability — selecting the appropriate quasi-experimental tool based on data structure and business context; for example, prioritizing Difference-in-Differences in policy change scenarios, or Regression Discontinuity when a natural threshold exists. Third, honest expression of uncertainty — observational analysis conclusions often depend on unverifiable assumptions; honestly disclosing these assumptions and quantifying their impact on conclusions is a basic requirement of responsible data analysis, and the long-term foundation of analytical credibility.

Conclusion

The discussion around In Praise of Observational Evidence reminds us that the value of scientific evidence should not be monopolized by any single methodology. RCTs are undeniably powerful, but observational evidence is an equally indispensable window through which humanity understands the world. At a time when causal inference methods are maturing rapidly and data is available at unprecedented scale, reaffirming the standing of observational evidence is not a concession to rigor — it is a respect for scientific pluralism.

Truly mature researchers do not blindly worship any single form of evidence. They know how to choose the most appropriate tool for the nature of the question at hand, and maintain a clear-eyed understanding of the strengths and limitations of each type of evidence.

Key Takeaways

• RCTs are not all-powerful; their "gold standard" status is a historical construction, originally designed for clinical drug evaluation
• Observational evidence has driven a vast number of major scientific discoveries throughout history, and occupies a unique ecological niche: real-world scale, long-term tracking, and rare-phenomenon discovery
• Simpson's Paradox and other confounding traps are fundamentally problems of causal graph structure — not purely statistical phenomena — and require causal thinking, not just more data, to address
• Quasi-experimental methods such as instrumental variables, regression discontinuity, and difference-in-differences provide rigorous methodological tools for observational causal inference, but each carries non-trivial prerequisite assumptions
• The FDA's RWE framework signals institutional re-acceptance of observational evidence, though data quality and review standards remain ongoing challenges
• Emerging fields like causal machine learning combine large-scale observational data with causal inference, opening new possibilities for estimating heterogeneous treatment effects
• A mature evidence view requires causal graph thinking, methodological matching ability, and honest expression of uncertainty — all three are essential