Inferential Statistics: Turning Sample Data into Population Knowledge

In the experimental workflow: for how inferential statistics is used when designing and running experiments, see Inference.

Collecting numbers is only half the battle. Inferential statistics let us generalize from a limited sample to a broader population and test whether observed patterns are likely due to chance or real phenomena.
From the humble t-test to multilevel Bayesian models, inferential tools underpin nearly every empirical claim in the social, health, and data sciences.

Table of Contents

1. Descriptive vs. Inferential Statistics
2. Sampling Distributions & the Central Limit Theorem
3. Confidence Intervals
4. Hypothesis Testing Framework
5. Effect Sizes & Practical Significance
6. Power Analysis & Sample Size
7. Common Pitfalls & Misinterpretations
8. Beyond p: Estimation & Bayesian Thinking
9. Transparent Reporting Standards
10. Key Takeaways
11. Further Reading

Descriptive vs. Inferential Statistics

Summarizing your own data with the Descriptive Statistics Calculator is the descriptive half of the table above.

Sampling Distributions & the Central Limit Theorem

• Sampling distribution: Probability distribution of a statistic (e.g., mean) over repeated random samples.
• Central Limit Theorem (CLT): Regardless of the underlying population shape, the sampling distribution of the mean approaches normal when n is large (≈30+), enabling Z & t approximations — explore the curve behind them with the Normal Distribution Calculator.
• Standard Error (SE): SD of sampling distribution; SE = σ / √n (or s / √n when σ unknown). Smaller SE ⇒ tighter inference.

Confidence Intervals

A 95% confidence interval (CI) is the range of values that would capture the true parameter in 95% of hypothetical repeated samples given the model.

• Formula (mean, σ unknown):

  x̄ ± t_{α/2, df} × (s / √n)

• Interpretation tips
  • Not the probability the parameter lies in the interval (frequentist).
  • Width shrinks with lower variance or larger n.
• Graphical display: Error bars, forest plots enhance clarity — or compute the interval directly with the Confidence Interval Calculator.

Hypothesis Testing Framework

1. Null (H₀): No effect / difference.
2. Alternative (H₁): Effect / difference exists (directional or two-tailed).
3. Test statistic: Quantifies departure from H₀ (e.g., t, F, χ², z) — run the t-test version with the t-Test Calculator.
4. Sampling distribution under H₀: CLT or exact test defines expected variation.
5. p-value: Probability of observing test statistic ≥ extreme if H₀ is true — get exact probabilities for common tests with the P-Value Calculator.
6. Decision rule: Compare p to α (commonly .05) or use CI / Bayes factor.

Rejecting H₀ ≠ proving H₁; it means data are inconsistent with H₀ beyond tolerable error.

Effect Sizes & Practical Significance

Report CIs for effect sizes to express precision, not just magnitude — convert between Cohen's d, Pearson's r, and η² with the Effect Size Calculator.

Power Analysis & Sample Size

• Power (1 – β): Probability of detecting a true effect (commonly 0.80).
• Inputs: α, expected effect size, variance, design (paired, cluster), directional test.
• Tools: G*Power, R *pwr*, the Power & Sample Size Calculator, Bayesian decision-theoretic methods.
• Sequential & adaptive designs: Re-estimate or stop early while controlling Type I error.

Common Pitfalls & Misinterpretations

The Type I/II Error Practice lets you check these distinctions hands-on.

Beyond p: Estimation & Bayesian Thinking

• Estimation approach: Focus on effect size + CI; ask “How big?” not “Whether.”
• Bayesian inference: Updates prior belief with data → posterior; outputs credible intervals, Bayes factors (compute them with the Bayes Factor Calculator), full distribution of uncertainty; intuitive probability statements (“There’s a 93% chance effect > 0”).
• Equivalence & non-inferiority tests: Show effect is sufficiently small (or not worse) rather than absent.

Transparent Reporting Standards

Include assumption checks, deviation disclosures, and raw effect plots.

Key Takeaways

1. Inferential statistics convert sample results into population claims using probability theory.
2. Sampling distributions, CIs, and hypothesis tests form the backbone; effect sizes & power add practical nuance.
3. Misuse (p-hacking, misinterpretation) erodes credibility—follow transparent, preregistered, assumption-aware practices.
4. Complement null-hypothesis tests with estimation, Bayesian methods, and robust descriptive visualizations for richer inference.

Methodology

Methodology: This topic is part of the experimental workflow covered in Experimentology. For the relevant methods and how they are evaluated in real experiments, see Inference and Estimation.

Further Reading

• Cumming, G. (2014). The New Statistics: Why and How.
• Wasserstein, R. & Lazar, N. (2016). “The ASA Statement on p-values.” The American Statistician.
• Kruschke, J. (2015). Doing Bayesian Data Analysis (2nd ed.).
• Greenland, S. et al. (2016). “Statistical tests, p values, confidence intervals…” European Journal of Epidemiology.
• Publication Bias — how the file drawer distorts the evidence these tests produce
• Chi-Square — the test for categorical associations