Reliability: Getting the Same Answer Every Time
In the experimental workflow: for how reliability is used when designing and running experiments, see Measurement.
A measure is reliable when it yields consistent results under consistent conditions.
Whether you’re weighing flour for bread, timing reaction speed, or scoring a depression scale, reliability tells you how much random error clouds the score. High reliability is a prerequisite for validity—an unreliable ruler can never be accurate.
Table of Contents
- Reliability vs. Validity
- Four Major Types of Reliability
- Estimating Reliability: Key Statistics
- Factors That Undermine Reliability
- How to Improve Reliability
- Acceptable Thresholds: Rules of Thumb
- Key Takeaways
- Further Reading
Reliability vs. Validity
| Concept | Core Question | Quick Analogy |
|---|---|---|
| Reliability | Is the measurement consistent? | Bathroom scale reads 72 kg every time today. |
| Validity | Is the measurement accurate? | Scale is calibrated so 72 kg is your true weight. |
Necessary but not sufficient: A reliable measure can still be invalid if it consistently misses the mark.
Four Major Types of Reliability
| Type | What It Checks | Typical Scenario | Statistic |
|---|---|---|---|
| Test-Retest | Stability over time | IQ test today vs. in two weeks | Pearson r or ICC |
| Internal Consistency | Consistency across items | 10-item anxiety scale | Cronbach’s α, McDonald’s ω |
| Inter-Rater (Inter-Observer) | Agreement among raters | Two clinicians score a video | Cohen’s κ, ICC |
| Parallel / Alternate Forms | Consistency across equivalent forms | Form A vs. Form B of SAT | Pearson r |
Split-half reliability is a special case of internal consistency (odd vs. even items).
Estimating Reliability: Key Statistics
| Statistic | Best For | Rule of Thumb |
|---|---|---|
| Cronbach’s α | Internal consistency (continuous items) | ≥ 0.70 acceptable; > 0.90 may indicate redundancy |
| Kuder-Richardson-20 (KR-20) | Dichotomous items (yes/no) | Same cut-offs as α |
| McDonald’s ω | When factor loadings vary | Often preferred to α |
| Intraclass Correlation (ICC) | Test-retest or inter-rater on interval data | ≥ 0.75 good; ≥ 0.90 excellent |
| Cohen’s κ / Fleiss’ κ | Inter-rater on categorical data | ≥ 0.60 substantial; ≥ 0.80 near-perfect |
Generalizability Theory extends these metrics to multiple facets (raters × items × occasions).
Factors That Undermine Reliability
- Poorly Worded Items – Ambiguity inflates error variance.
- Heterogeneous Construct – Too many unrelated facets lower internal consistency.
- Small Sample or Few Items – Limited variance or item pool depresses coefficients.
- Testing Conditions – Noise, fatigue, rapport shifts between sessions.
- Rater Drift – Observers change standards over time without calibration.
How to Improve Reliability
| Strategy | Targeted Problem |
|---|---|
| Refine Items – use clear, specific language | Ambiguity |
| Increase Item Count (Spearman-Brown prophecy) | Low internal consistency |
| Standardize Administration – scripts, quiet rooms | Situational noise |
| Rater Training & Calibration Sessions | Inter-rater disagreement |
| Pilot Testing – detect ceiling/floor effects | Restricted variance |
| Automate Scoring / Use Wearables | Human error |
Remember: adding items boosts α, but only if new items tap the same construct.
Acceptable Thresholds: Rules of Thumb
| Context | Minimum Recommended |
|---|---|
| Early research / exploratory scales | α ≈ 0.70 |
| Clinical diagnosis | α or ICC ≥ 0.90 |
| Classroom quizzes (low stakes) | α ≈ 0.60 can suffice |
| High-stakes certification exams | α ≥ 0.95 |
Thresholds are guidelines, not absolutes—interpret in light of construct breadth and decision stakes.
Key Takeaways
- Reliability = consistency; it places an upper limit on validity.
- Four main flavors—test-retest, internal consistency, inter-rater, parallel forms—ask different consistency questions.
- Use appropriate statistics (Cronbach’s α, ICC, κ) and design tweaks (clear items, rater training) to maximise dependability.
- Evaluate reliability continuously; coefficients can change with population, context, and item revisions.
Methodology
Methodology: This topic is part of the experimental workflow covered in Experimentology. For the relevant methods and how they are evaluated in real experiments, see Measurement.
Further Reading
- Kline, P. (2015). A Handbook of Test Construction.
- DeVellis, R. (2021). Scale Development: Theory and Applications (5th ed.).
- Shrout, P. & Fleiss, J. (1979). “Intraclass correlations.” Psychological Bulletin.
- Cronbach, L. & Shavelson, R. (2004). “My current thoughts on G-theory.” Measurement.
- Reliability & Validity Practice — interactive drills for the measurement concepts on this page
- Cronbach's Alpha Calculator — compute internal-consistency reliability from your item data
References
- Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.