Meta-analysis
The basics of evidence synthesis
A single study is noisy; synthesis across studies is stronger.
- How not to synthesize — vote-counting ("k of N significant") ignores effect sizes and power.
- Fixed-effects meta-analysis — assumes one true effect; averages studies but underestimates uncertainty when studies differ.
- Limitations of fixed-effects — real effect sizes vary across contexts; fixed-effects can't represent that.
- Random-effects meta-analysis — models a distribution of true effects across studies, giving a more honest interval when heterogeneity exists.
- Single-paper meta-analysis and pooled analysis — combining conditions within one paper to raise power.
Bias in meta-analysis
- Within-study biases — each study may carry its own design or measurement bias (see Measurement and Design).
- Across-study biases — differences in samples, settings, and analyses across studies.
- Publication bias — significant results are more likely published, skewing the pooled estimate. Tools like p-curve and funnel plots quantify it.
- Across-study bias correction — methods that adjust for publication bias, with caveats.
- Garbage in, garbage out? — meta-analyzing weak studies doesn't create a strong conclusion; synthesis quality is bounded by study quality.
Key terms
- Random-effects model — effect sizes vary across studies; estimates a mean and spread.
- Publication bias — selective publication of significant results.
- Forest plot — a visual of each study's effect and the pooled estimate.
Further reading
- Previous: Visualization · Next: Conclusion · Hub: Experimentology
- Why synthesis is needed: Replication
- Plan for power: Sampling
- Publication Bias — the file-drawer problem behind the studies a synthesis never sees.
- Inferential Statistics — the tests pooled in meta-analysis.
- Modeling — meta-analytic models as statistical models.
- Effect Size Calculator — compute standardized effect sizes to pool across studies
References
- Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.