What Real-World Evidence Can and Cannot Tell You
Real-world evidence is being used to support regulatory decisions in ways that would have been unthinkable a decade ago. That is either exciting or alarming, depending on how well you understand its limits.
In 2016, the 21st Century Cures Act in the United States instructed the FDA to develop a framework for using real-world evidence to support regulatory decisions, including drug approvals. Since then, the FDA has approved several interventions partly on the basis of real-world data, and the European Medicines Agency has moved in a similar direction through its DARWIN EU initiative.
This is a significant shift. For decades, the randomised controlled trial was the unambiguous gold standard for regulatory evidence, and real-world data was considered useful for hypothesis generation but not for causal inference. The new frameworks represent a pragmatic recognition that RCTs have real limitations, they are expensive, slow, and often enrol populations that do not reflect the patients who will actually use the treatment, and that routinely collected healthcare data has grown to a scale and quality where it can sometimes substitute.
But the shift also carries risks that are worth understanding clearly. Real-world evidence can tell you a great deal about what happens in practice. It is much harder to use it to establish why.
What real-world evidence actually is
Real-world evidence is derived from real-world data, which is any data collected outside of a controlled trial setting. This includes electronic health records, administrative claims databases, disease registries, patient-reported outcome surveys, wearable device data, and pharmacy dispensing records. The data exists because patients received care, not because a trial was designed to collect it.
The appeal is obvious. A large national claims database might contain records for millions of patients across years of follow-up, in populations that include the elderly, patients with multiple comorbidities, and patients from diverse socioeconomic backgrounds, populations that clinical trials routinely exclude. If you want to know how a treatment performs in the real world, real-world data is, almost by definition, where you should look.
The challenge is that real-world data was not collected to answer your question. It reflects the decisions of thousands of clinicians treating thousands of patients, each influenced by factors that the data may or may not capture. The resulting confounding, the systematic differences between patients who received one treatment versus another, is the central methodological problem in observational research.
Confounding is not a detail problem
In a well-designed randomised trial, randomisation distributes known and unknown confounders roughly equally across treatment groups. This is the specific thing that allows causal inference: the only systematic difference between groups is the treatment itself.
In a real-world dataset, patients who received treatment A and patients who received treatment B are different in ways that the database may partially capture and partially not. Clinicians prescribe based on clinical judgement, which reflects the patient's full presentation, including factors like disease severity, functional status, comorbidity burden, and the clinician's assessment of likely tolerability. These factors influence both treatment choice and outcome, which means any observed difference in outcomes between treated and untreated patients is a mixture of the treatment effect and the confounding effect of the factors that drove treatment selection.
Sophisticated analytical methods exist to address this: propensity score matching, inverse probability weighting, instrumental variable analysis, difference-in-differences designs. These methods reduce confounding but do not eliminate it. They adjust for the confounders that were measured. The confounders that were not measured, which in healthcare data are substantial, remain. This is called residual confounding, and it is the central reason why observational studies and RCTs continue to produce discordant results in some areas.
Where real-world evidence genuinely adds value
Despite these limitations, there are research questions where real-world evidence is not just useful but essential.
Long-term safety signals are the clearest case. RCTs are typically too short and too small to detect adverse events that occur with low frequency over long time horizons. Post-approval pharmacovigilance using real-world data has identified safety signals, including cardiac events, hepatotoxicity, and rare immune reactions, that clinical trials missed. Some of these signals have led to market withdrawals or label changes that directly affected patient safety. This is real-world evidence working as it should.
Comparative effectiveness research in practice populations is another genuine strength. An RCT comparing two treatments in a selected patient population tells you about efficacy under controlled conditions. A well-designed observational study in a broad electronic health record dataset can tell you how those treatments perform when prescribed to the kinds of patients clinicians actually see, which is often a more useful answer for clinical decision-making.
Rare diseases and paediatric populations, where traditional trials face substantial recruitment challenges, are areas where real-world evidence fills a gap that RCTs simply cannot close within any reasonable timeframe.
The regulatory trajectory and its implications
FDA guidance published between 2019 and 2023 has progressively expanded the evidentiary role of real-world evidence, including for post-approval effectiveness studies and, in some cases, as supplementary evidence for approval decisions. Japan's Pharmaceuticals and Medical Devices Agency (PMDA) has developed parallel frameworks for RWE use in regulatory submissions, and China's National Medical Products Administration (NMPA) published its own RWE guidelines in 2020. The COVID-19 pandemic accelerated this trajectory globally, as regulators used real-world data to assess vaccine effectiveness at speeds that conventional trial designs could not match.
The implication for clinical researchers and pharmaceutical developers is that real-world data capabilities are becoming a core competency rather than a specialised niche. Research teams that cannot design, execute, and defend observational studies using large administrative datasets are increasingly at a disadvantage in regulatory interactions and funding competitions.
This has created a market for methodological expertise in causal inference from observational data, and a corresponding risk that organisations without that expertise will reach for real-world evidence as a faster or cheaper alternative to trials without fully understanding what it can and cannot establish.
What this means for how you synthesise the evidence
One practical consequence of the growing volume of real-world evidence in the literature is that systematic reviewers and researchers synthesising evidence now need to engage with a more methodologically heterogeneous body of work than was typical even five years ago. A search on a treatment's effectiveness might return a mix of RCTs, registry studies, claims-based observational analyses, and single-arm cohort studies, each with different risks of bias and different implications for how much weight to place on the findings.
Making sense of that landscape, understanding where the RCT and real-world evidence converge and where they diverge and why, is a non-trivial task that requires both methodological knowledge and a comprehensive view of the evidence. Missing a key observational study that contradicts the trial evidence, or overlooking the reason the discordance exists, leads to synthesis conclusions that are at best incomplete.
This is precisely the kind of problem that AI-assisted evidence mapping is beginning to address at NousLab, helping researchers get a faster and more complete view of a heterogeneous evidence landscape before they start synthesis. See how it works in practice.
