Replication
Reproducibility vs. replication
These two words are often confused but differ:
- Reproducibility — can someone re-run your analysis on your data and get the same numbers? (A coding/transparency issue.)
- Replicability — can an independent study, following the same procedure, recover a similar result in new data? (A scientific-robustness issue.)
The Open Science Collaboration (2015) famously estimated that only about 36% of the 100 replicated psychology findings reached significance, and that the replicated effect sizes were on average about half the size of the originals—the "replication crisis."
Conceptual frameworks for replication
- Direct replication — repeat the study as closely as possible to check the result.
- Conceptual replication — test the same idea with a different method, sample, or stimulus, checking the underlying phenomenon.
- "Small Telescopes" — a framework for judging whether a replication had enough power to detect an effect of plausible size, so a null result is interpretable rather than ambiguous.
The metascience of replication studies which practices produce reliable knowledge.
Causes of replication failure
When an effect fails to replicate, suspect:
- Context, moderators, and expertise. The original may have held only under specific conditions or with specific populations.
- Analytic flexibility. Choosing analyses after seeing the data ("researcher degrees of freedom") can manufacture seemingly significant results that don't survive pre-registered confirmation. This is the single biggest methodological culprit—see Preregistration.
The personal and career consequences for the original authors are real, which is why replication should be normalized, not punitive.
Replication, theory building, and open science
Replication and theory are reciprocal: a failed replication can refine a theory; a good theory predicts when replication should succeed. Open science practices—open data, materials, analysis code, and preregistration—make both reproducibility and replicability achievable and let the field self-correct.
Key terms
- Direct replication — same procedure, new data.
- Conceptual replication — same idea, different method.
- Analytic flexibility — post-hoc choices in analysis that inflate false positives.
Further reading
- Previous: Theories · Next: Ethics · Hub: Experimentology
- The fix: Preregistration
- Pool the evidence: Meta-analysis
- Inferential Statistics — why p-values are fragile without preregistration.
- Experimental Design — designs that support clean replication.
- Stroop Effect — a robust effect that has replicated widely.
References
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.