Validity: The Gold Standard of Good Measurement and Inference
In the experimental workflow: for how validity is used when designing and running experiments, see Measurement.
When researchers talk about validity, they’re asking a deceptively simple question:
Are we really measuring (or concluding) what we think we are?
Validity is the foundation of credible science, robust assessment, and evidence-based decision-making. Without it, a test score, survey item, or experimental finding may look precise yet still miss the mark.
Table of Contents
- Validity vs. Reliability
- Measurement Validity: The Big Three
- Internal & External Validity in Experiments
- Threats to Validity
- Establishing and Evaluating Validity
- Best Practices for Boosting Validity
- Key Takeaways
- Further Reading
Validity vs. Reliability
| Concept | Core Question | Analogy |
|---|---|---|
| Reliability | Is the measurement consistent? | Bathroom scale gives same weight each time. |
| Validity | Is the measurement accurate? | Does that weight match your true weight? |
Reliability is necessary but not sufficient for validity: a bathroom scale that’s mis-calibrated by 5 kg is reliable yet invalid.
Measurement Validity: The Big Three
| Type | What It Asks | Typical Evidence |
|---|---|---|
| Content Validity | Does the test cover the full domain? | Expert panels, blueprint matrices. |
| Criterion Validity | Does the score correlate with relevant outcomes? | Concurrent (with gold standard) or predictive (future behavior). |
| Construct Validity | Does the test behave as the theory predicts? | Convergent & discriminant correlations, factor analysis, known-groups contrasts. |
Modern view: Construct validity is the umbrella, with content and criterion evidence feeding into it (Standards for Educational & Psychological Testing, 2014).
Quick Glossary
- Face validity – It looks right to stakeholders (weakest form).
- Ecological validity – Lab findings generalize to real-world contexts.
- Incremental validity – Adds unique predictive power over existing measures.
Internal & External Validity in Experiments
| Domain | Key Question | Boosters |
|---|---|---|
| Internal Validity | Is the causal inference warranted? | Random assignment, control groups, blinding, pre-registration. |
| External Validity | Do results generalize across people, settings, time? | Representative samples, replication, field studies. |
A tightly controlled lab study may nail internal validity yet sacrifice external validity; field studies flip the trade-off.
Threats to Validity
| Threat | Affects | Example |
|---|---|---|
| Selection Bias | Internal | Treatment and control differ on motivation. |
| History & Maturation | Internal | Participants change over time unrelated to treatment. |
| Testing & Instrument Decay | Internal | Pre-test sensitizes responses; equipment drifts. |
| Construct Under-representation | Measurement | Math test omits geometry. |
| Common-Method Bias | Measurement | Same respondent, same scale inflates correlations. |
| Range Restriction | Criterion | Only high performers hired → low test-job correlation. |
Establishing and Evaluating Validity
- Theoretical Rationale – Define the construct and its boundaries.
- Test Blueprinting – Ensure items sample the entire domain.
- Pilot Testing & Expert Review – Check clarity and relevance.
- Statistical Analyses
- Exploratory & Confirmatory Factor Analysis for structure.
- Correlation / regression with external criteria.
- Multitrait–Multimethod matrices for construct validation.
- Cross-Validation – Replicate on independent samples.
- Consequence Analysis – Examine intended and unintended uses.
Best Practices for Boosting Validity
- Triangulate Evidence: Combine qualitative (expert judgement) and quantitative (statistics) sources.
- Use Established Instruments: When possible, adopt or adapt tools with known validity.
- Mitigate Confounds: Randomize, counterbalance, or statistically control.
- Pilot & Revise: Catch ambiguous items or procedural glitches early.
- Report Transparently: Detail sampling, measures, manipulation checks; preregister hypotheses.
- Seek Diverse Samples: Enhances external and cross-cultural validity.
Key Takeaways
- Validity is about truthfulness—does the measure or inference accurately reflect reality?
- Content, criterion, and construct evidence weave together to support measurement validity, while internal and external validity safeguard causal generalizations.
- Threats abound, but careful design, triangulation, and transparent reporting can defend against them.
Methodology
Methodology: This topic is part of the experimental workflow covered in Experimentology. For the relevant methods and how they are evaluated in real experiments, see Measurement.
Further Reading
- American Educational Research Association et al. (2014). Standards for Educational and Psychological Testing.
- Cook, T. & Campbell, D. (1979). Quasi-Experimentation: Design & Analysis Issues.
- Messick, S. (1995). “Validity of Psychological Assessment: Validation of Inferences From Persons’ Responses.” American Psychologist.
- Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference.
- Reliability & Validity Practice — interactive drills for distinguishing measurement concepts
References
- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302.