How to Design an Observational Study That Survives Peer Review
Observational studies are the backbone of real-world evidence - and the source of most retractions. Getting the design right before data collection is the difference between credible findings and a desk reject.
In the early 1980s, a series of case-control studies suggested that coffee consumption was associated with pancreatic cancer. The studies were widely reported. Subsequent cohort studies, with better control for smoking as a confounder, found no such association. Coffee was not the problem; the studies were. The finding collapsed not because of fraud but because of design choices that seemed reasonable at the time and were not.
Choosing the Right Study Design
The first decision in observational research is choosing the appropriate design for the question. Cohort studies follow exposed and unexposed participants forward in time and are well-suited to estimating incidence and studying rare exposures with multiple outcomes. Case-control studies identify cases and controls and look backward at exposure history - they are efficient for rare outcomes and fast to execute, but prone to recall bias and difficult to design well. Cross-sectional studies capture exposure and outcome at a single point in time and are useful for prevalence estimates but cannot establish temporality.
The most common design error is choosing a cross-sectional study for a question that requires a cohort design, or a case-control design for a question where the exposure itself is rare enough to make finding an adequate control group difficult. These decisions should be made before data collection, not after.
Confounding by Indication
In pharmacoepidemiology, confounding by indication is the most persistent threat to observational study validity. When a drug is prescribed to patients who are sicker than those who are not prescribed it, any comparison of outcomes between the two groups will partly reflect the underlying disease severity rather than the drug's effect. Patients prescribed anticoagulants are at higher baseline risk of bleeding; patients prescribed statins after a cardiac event are at higher risk of recurrence. Naive comparisons will make treatments look harmful or ineffective when they are not.
Methods to address confounding by indication include propensity score methods (matching, stratification, inverse probability weighting), active comparator designs that compare the drug of interest to another drug with similar indications, and restriction to patients at the margin of the prescribing decision. No method eliminates unmeasured confounding entirely - the honest observational study acknowledges what its design can and cannot control for.
Defining the Exposure Correctly
Exposure definition is where many otherwise well-designed observational studies introduce silent bias. In database studies using electronic health records or claims data, the exposure window, the definition of new use versus prevalent use, and the handling of gaps in treatment all require explicit decisions. New-user designs - which restrict analysis to patients initiating the treatment under study - avoid the prevalent-user bias that affects studies including patients already established on treatment, who are by definition survivors and tolerators of the treatment.
The exposure definition should be specified in the protocol before data access, not adjusted after looking at the data. If the exposure definition changes after the analysis is run, the study has moved from confirmatory to exploratory without acknowledging it.
The STROBE Checklist
The STROBE checklist (Strengthening the Reporting of Observational Studies in Epidemiology) provides a 22-item framework for reporting cohort, case-control, and cross-sectional studies. It is not a quality assessment tool - it is a reporting standard - but reviewers use it as a proxy for study quality, and a manuscript that fails to address its items clearly will face questions about whether the omissions reflect reporting gaps or design gaps.
Key STROBE items that manuscripts most commonly handle poorly include: the explicit description of how potential confounders were identified and addressed; the handling of missing data; the description of sensitivity analyses; and the honest acknowledgement of what the study design can and cannot establish. Writing the limitations section as an afterthought, rather than as a genuine engagement with the design's constraints, is a reliable way to attract the concern of thorough reviewers.
How Evidence Synthesis Changes Study Design
One of the least practised but most valuable steps in observational study design is conducting a systematic review of existing evidence before finalising the protocol. This serves multiple purposes. It identifies effect size estimates that can inform power calculations without assuming optimistic numbers. It reveals the confounders that prior studies have identified as important, which should be measured in the new study if possible. It surfaces the methodological weaknesses of existing studies, which the new study can be explicitly designed to avoid. And it identifies whether the question has already been answered adequately - a genuine possibility that is worth knowing before investing in a new study.
At NousLab, we work with teams that are designing observational studies for regulatory and HTA purposes, where the quality bar for design and reporting is higher than in academic publishing. The time spent on a scoping review or rapid evidence map before the study begins consistently improves the quality of the protocol that emerges. See how NousLab supports evidence-driven study design.
