Measurement

Reliability

A measure is reliable when it gives consistent results—across time (test–retest), items (internal consistency), or raters (inter-rater). This approach uses a children's vocabulary measure as a worked example: a good test correlates with itself and with related constructs.

• Measurement scales — nominal, ordinal, interval, ratio; the scale determines which statistics are meaningful.
• Measuring reliability — Cronbach's α (compute it with the Cronbach's Alpha Calculator), test–retest correlations, intraclass correlations.
• Reliability paradoxes — a measure can be reliable yet meaningless; and some "unreliable" measures are just noisy-but-valid. Reliability is necessary, not sufficient.
• Practical advice — more items usually improve reliability; pilot your measure before the real study.

Validity

A measure is valid when it measures what you claim.

• Validity arguments in practice — construct, content, criterion, and (for experiments) internal validity. Build a reasoned case, don't assume.
• Avoid questionable measurement practices — dropping items post-hoc to "improve" α, or defining a construct by its measure, quietly undermines validity.

How to select a good measure?

• Simple vs. complex behaviors — simple behaviors (reaction time) are easy to measure; complex constructs (intelligence, attachment) need careful instrument choice.
• Implicit vs. explicit behaviors — implicit measures (e.g., reaction-time tasks) can bypass self-report bias but bring their own validity questions.

The temptation to measure lots of things

Collecting many measures "just in case" invites p-hacking and multiverse instability. Pre-specify the measures that matter.

Key terms

• Reliability — consistency of a measure.
• Validity — accuracy of what a measure claims to assess.
• Construct validity — the measure truly reflects the theoretical construct.

Further reading

• Previous: Models · Next: Design · Hub: Experimentology
• The outcome you measure: Experiments
• Report it well: Visualization
• Reliability — the foundational treatment in Research Methods.
• Validity — types and threats to validity.
• Likert Scale — a common measurement instrument.
• Implicit Association Task — an implicit measure where measurement validity is the central question.
• Psychomotor Vigilance Task — reaction time and lapses as continuous dependent variables.
• Spatial Span — capacity as a working-memory metric.
• Reliability & Validity Practice — interactive drills for measurement concepts
• Likert Scale Generator — build your own summated rating scale
• Cronbach's Alpha Calculator — compute internal-consistency reliability from item data