Understanding Bias in Clinical Research: The Ones That Actually Matter
Every researcher learns about bias in methods training. Most learn a list of names. What is harder to teach is how to recognise them in your own work before peer review does it for you.
Bias is one of those concepts that researchers learn early and apply inconsistently throughout their careers. The methods textbook version, a list of named biases with definitions and examples, is easy to understand and not very useful in practice. The useful version is the ability to look at a study design and recognise, before the data are collected, where the systematic errors are likely to enter and how large their effects are likely to be.
That skill is harder to acquire than the textbook version suggests. Partly because it requires genuine familiarity with a field's conventions and failure modes, not just abstract methodological knowledge. Partly because bias feels hypothetical until the results are in, at which point it is too late to redesign the study.
We spend time thinking about this problem at NousLab because the literature that researchers synthesise using the platform is biased in specific, patterned ways, and understanding those patterns changes how you read and weight the evidence. The biases in the existing literature are not randomly distributed. They are systematic, and they systematically affect which questions appear answered when they are not.
Selection bias: the one that compounds
Selection bias enters when the people who are studied are systematically different from the people the findings are meant to apply to. In clinical trials, this is usually obvious: inclusion and exclusion criteria define a study population that may differ substantially from clinical practice. The issues are acknowledged in the limitations section and mostly ignored in how the findings are applied.
The less obvious form occurs in observational research when the decision to seek care, or to be recorded in the data system, is related to the outcome being studied. A database of hospital admissions systematically excludes everyone whose condition was managed at home or in primary care. A registry of cancer patients enrolled through academic medical centres overrepresents urban, educated, higher-income patients. The findings from these samples are not wrong, they are right about the people who were studied. The problem is the scope of the claims made from them.
Selection bias compounds when it interacts with other forms of bias. A study that both selects a non-representative sample and measures outcomes imperfectly produces errors that cannot be separated at the analysis stage. Each form of bias can be partially addressed individually; together, they produce confounding that no statistical adjustment can fully disentangle.
Information bias and the measurement problem
Information bias occurs when the measurement of exposure, outcome, or both is systematically inaccurate in a way that is related to the other variable. The most common form in clinical research is differential misclassification: the measurement error is not random but patterned, affecting exposed and unexposed, or case and control, groups differently.
Recall bias is the canonical example. In a case-control study, patients who experienced a severe outcome are more likely to recall and report prior exposures than controls who did not. The bias is not intentional, it is a natural consequence of how memory works. But it systematically inflates apparent associations between exposure and outcome in retrospective designs.
Electronic health records have introduced new forms of information bias that are only beginning to be well-characterised. Diagnostic coding varies between clinicians and institutions. Prescribing data reflects what was prescribed, not what was taken. Dates of diagnosis in EHR systems may reflect when a diagnosis was recorded rather than when it occurred. None of these errors are random. They are patterned by the organisation, the clinician, the patient's behaviour, and the disease's presentation, all of which may be related to the outcomes of interest.
Confounding: the one that is hardest to see in your own work
Confounding occurs when the apparent relationship between an exposure and an outcome is actually explained by a third variable that is related to both. It is well understood at the conceptual level and persistently underestimated in practice, particularly when the confounders are not measured.
The insidious thing about unmeasured confounding is that it is invisible in the results. A well-executed analysis with a beautiful table of baseline characteristics showing balance across groups may still be substantially confounded by variables that were never collected. Presenting a propensity score-matched analysis does not eliminate confounding by unmeasured variables. It eliminates confounding by the variables that were measured and included in the propensity model.
One underused approach for assessing unmeasured confounding is the E-value, developed by VanderWeele and Ding, which estimates how strong an unmeasured confounder would need to be to explain away an observed association. An effect estimate with a low E-value is more vulnerable to unmeasured confounding than one with a high E-value. It does not tell you whether confounding exists, but it quantifies how much unmeasured confounding would be needed to invalidate your finding, which is genuinely useful for interpreting results. The original paper by VanderWeele and Ding in Annals of Internal Medicine is worth reading in full.
Reporting bias in the literature: the meta-problem
The biases above occur within studies. Reporting bias occurs across studies, and it affects the literature that researchers synthesise to design their own work.
Studies with positive or statistically significant results are published more readily, cited more often, and translated into clinical practice more quickly than studies with null or ambiguous results. This is well-documented. What is less often discussed is that reporting bias is not just about what gets published but about what gets written up at all. Negative results frequently sit in researchers' file drawers, never submitted for publication, representing a systematic hole in the indexed evidence base.
When you synthesise the literature to design a new study, you are working with a sample of the evidence that has been filtered by this process. The picture you see is not representative of what has been tested. It is a curated selection tilted toward findings that confirmed their hypotheses, often with inflated effect sizes due to selective outcome reporting within studies.
The practical implication is that when you find strong apparent consensus in the literature, it is worth asking whether that consensus reflects the totality of what has been tested or whether it reflects publication patterns. This is particularly important in areas with commercial involvement, where the gap between what has been tested and what has been published is well-documented and consequential.
Building bias awareness into the design stage
The most useful time to think about bias is before data collection, not during analysis. By the time results are in, the options for addressing design-level biases are limited to analytical workarounds and transparency in reporting. Neither fully compensates for a bias that was built in at the design stage.
A structured pre-registration process, including a bias risk assessment component, forces this thinking onto the agenda at the right time. Registering a study protocol on ClinicalTrials.gov or the UMIN Clinical Trials Registry in Japan commits the team to a pre-specified analysis plan before data collection begins, which substantially limits the scope for post hoc analytical decisions. The STROBE guidelines for observational studies and the Cochrane Risk of Bias 2 tool for trials are useful frameworks, not just for evaluating other people's work but for anticipating where your own design is vulnerable before you commit to it.
Understanding where bias enters the literature is also foundational to how you read and synthesise existing evidence before designing your own study. See how NousLab helps research teams map the evidence landscape with bias patterns in mind.
