Inference

Sampling variation

Even a perfect experiment gives a different estimate each time you resample, because of random variation. Two key ideas:

Standard errors quantify how much an estimate would bounce around across repeated samples.
The central limit theorem tells us that, with enough data, the distribution of estimates becomes approximately normal—making errors and intervals computable.

Inference is the art of distinguishing a real pattern from this random bounce.

Three frameworks coexist; know which you're using:

p-values — the probability of data at least this extreme if there were no effect. A small p is evidence against the null, but says little about the effect's size — turn a test statistic into a probability with the P-Value Calculator.
Bayes factors — the ratio of evidence for one model vs. another; can quantify support for the null, not just against it (compute one from a t statistic with the Bayes Factor Calculator).
The Neyman–Pearson approach — pre-commit to a rule (significance level α, power) for deciding between acting as if an effect exists or not; controls long-run error rates.

Problems with p-values. p depends on sample size, is not the probability the hypothesis is true, and is routinely misinterpreted. "p < .05" is not "true."
What framework to use? This approach leans toward estimation with uncertainty (effect sizes + intervals) and treats p-values as one tool among several, not the goal.

Confidence intervals give a range that, under repeated sampling, covers the true value at a set rate. They communicate both location and precision—exactly the preferred currency here, and the Confidence Interval Calculator builds them from sample statistics.