The File Drawer Effect Is Not the Whole Story: Understanding Publication Bias in Psychology

In psychology, the file drawer effect is often invoked as a convenient explanation for why so many published findings turn out to be fragile or irreproducible. The idea is simple: journals prefer novel, statistically significant results, so studies that fail to find significance end up "locked in a drawer," unseen and unpublished. What we read in journals, therefore, is a systematically biased subset of reality—mostly positive findings.
This diagnosis is not wrong. The file drawer effect is real, and it is a form of publication bias.
However, treating the two as interchangeable obscures a deeper and more damaging problem. Distinguishing them is not pedantic—it is essential for understanding how entire research fields can drift away from the truth while appearing healthy and productive.

Beyond the File Drawer: What Publication Bias Really Looks Like

If publication bias were only about journals rejecting null results, the damage would be limited. Researchers enjoy considerable analytical flexibility—what is often called researcher degrees of freedom—so many studies can be nudged into statistical significance anyway. In such a world, relatively few studies would truly remain unpublished.
The more serious issue is this:
Psychology strongly disfavors direct replication.
Psychologists rarely repeat previous experiments verbatim. Even conceptual replications—studies that test the same idea using slightly modified methods—are far more common than exact replications. As long as the theory sounds consistent, methodological drift is tolerated, even encouraged.
This tendency has been criticized for decades. Richard Feynman noticed it. Paul Meehl noticed it. Yet it persisted, quietly shaping incentives and norms.
The result is not merely selective reporting—it is a systemic failure to correct errors.

A Chain Reaction of Error

When these elements combine, a predictable dynamic emerges. It unfolds roughly as follows:

1. A statistically underpowered researcher has a novel idea and designs a weak study.
2. Through analytical flexibility, a statistically significant result is obtained.
3. The estimated effect size is surprisingly large, drawing attention (a classic Type M error, or magnitude inflation).
4. Other researchers follow up—not by directly replicating the original study, but by slightly modifying it.
5. Many also obtain significant results, again aided by researcher degrees of freedom.
6. Inflated effect sizes continue to appear.
7. The few studies that fail to support the hypothesis are never published—or never even submitted—so the community remains unaware of the negative evidence (the file drawer effect).
8. Steps 4–7 repeat.
9. The field appears vibrant, coherent, and empirically well-supported.
10. Eventually, someone asks: “What if we directly replicate the original experiment?”
    This pattern is widely believed to describe what happened in social priming research. The original studies—some with sample sizes as small as 30—were accepted with confidence. Even Daniel Kahneman expressed strong belief in the findings at the time. It is difficult to explain this episode by fraud or bad faith alone. Many sincere, competent researchers genuinely believed the effects were real.
    This model is not new. Paul Meehl articulated a similar concern as early as 1967. What is striking is not that he was wrong—but that he was ignored.

Meehl’s Warning, Revisited

Meehl was widely respected. His critiques of null hypothesis significance testing were well-known and heavily cited. Yet decades later, he wrote with visible frustration:

“Despite my having received more than 1,000 reprint requests for that article in the first year after its appearance, I cannot discern that it had more impact on research habits in soft psychology than did Morrison and Henkel. Our graduate students typically plan and write their doctoral dissertations in blissful ignorance that ‘the significance test controversy’ even exists…”
This was not a minor methodological quibble. Meehl believed the problem was grave.
Today’s replication crisis suggests he was right.

How Should We Respond?

“While it is easy to lie with statistics, it is even easier to lie without them.”
— Frederick Mosteller

The Productive Response

For outsiders—students, readers, and users of psychological knowledge—the key is not cynicism, but structure.
Leading psychologists were not blindsided by the crisis. Many had anticipated it and built safeguards into how theory should be interpreted. A good example appears in David Buss’s Evolutionary Psychology, where an entire early chapter is devoted to methodology—something common in Western textbooks and rare in many others.
Buss implicitly adopts Imre Lakatos’s philosophy of science, distinguishing between:

• a hard core (fundamental assumptions),
• a protective belt (auxiliary theories),
• and specific testable hypotheses.
  When a hypothesis fails, not everything collapses. Some elements are revised, others retained. After the replication crisis, many specific claims in psychology deserve skepticism or revision. Some mid-level theories should be updated. But the hard core—such as evolutionary theory itself—remains intact.
  This framework allows us to learn from failure without discarding the entire discipline.

The Unproductive Response

The most common reaction, however, is nihilism: if so much was wrong, then everything must be wrong. This response is emotionally satisfying and cognitively cheap—but ultimately unhelpful.
Psychology is a soft science, but that does not mean it is unscientific. Its problems are methodological, institutional, and cultural—not proof that the human mind is unknowable.

A Final Reflection

The replication crisis did not emerge because psychologists are uniquely careless or dishonest. It emerged because well-intentioned people operated within incentive structures that rewarded novelty, discouraged replication, and confused statistical significance with truth.
Understanding the distinction between the file drawer effect and publication bias helps us see the crisis more clearly—and respond to it more intelligently.
At kuakua, we believe psychological knowledge is still worth engaging with—carefully, critically, and with humility. Not everything needs to be believed. But much can still be understood.

Related Resources

• Inferential Statistics — the tests whose results publication bias distorts
• Meta-analysis workflow — synthesizing evidence across studies
• Replication workflow — reproducibility and open science

Methodology

Methodology: This topic is part of the experimental workflow covered in Experimentology. For how publication bias is handled when evidence is synthesized, see Meta-analysis.

References

• Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.