Preregistration

Lost in a garden of forking paths

Every dataset admits countless legitimate analyses. Choosing among them after seeing the data—data-dependent analysis—inflates false positives, because the path is picked to "work."

Only human — cognitive biases and skewed incentives (publish-or-perish, significance = publication) make post-hoc choices tempting even for honest researchers.
HARKing — "Hypothesizing After the Results are Known": presenting a post-hoc finding as if it were the planned hypothesis.

Together these are the researcher degrees of freedom problem, a primary driver of the replication crisis.

Reducing risk with preregistration

Preregistration is posting your plan—hypotheses, design, and analysis—before seeing the data. It:

Reduces bias by fixing analysis choices in advance.
Increases transparency into your reasoning.
Calibrates confidence — confirmatory results (planned) are weighted differently from exploratory ones.

Preregistration is one tool in a toolbox (alongside registered reports and analysis plans) for addressing researcher degrees of freedom.

How to preregister

State the research question, the design, the exact analyses (including which tests, which exclusions), and the sample-size plan. Keep it specific enough to bind your future self, but allow exploratory analyses to be labeled as such.

Key terms

Preregistration — a time-stamped plan filed before data analysis.
Researcher degrees of freedom — the many analysis choices that can be tuned post-hoc.
HARKing — presenting a post-hoc hypothesis as a priori.

References

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. PNAS, 115(11), 2600-2606.