Experimentology

Experimentology is the set of practices, findings, and approaches that enable the construction of robust, precise, and generalizable experiments.

The central thesis of this approach is simple but demanding:

Experiments are intended to make maximally unbiased, generalizable, and precise estimates of specific causal effects.

Everything below follows from that thesis. Use this section as a methodology hub: it explains why a Stroop task is built the way it is, how measurement reliability and validity are judged, why sampling strategy changes what you can claim, and how to report results so others can reproduce them.

The four cross-cutting themes

Every chapter returns to four ideas. Keep them in mind as you read:

  • Transparency — other researchers must be able to determine exactly what you did. Preregistration, open materials, and analysis scripts create this transparency.
  • Precision — plan experiments around "what causal effect do I want to measure?" and choose design, sampling, and analysis to maximize the precision of that estimate (effect sizes and confidence intervals, not just significance).
  • Bias reduction — beyond random error, measurements carry systematic error. Watch for sources of bias at every stage.
  • Generalizability — any single experiment covers only a slice of the conditions where a behavior occurs. Consider generalizability from stimulus selection to reporting.

These themes are referenced throughout the chapter pages via "Theme links" at the end of each.

Table of Contents

Part 1 — Foundations

  1. Experiments — what experiments are for, causality vs. observation, the anatomy of a randomized experiment.
  2. Theories — psychological constructs, theory testing, falsificationism, formalizing theories.
  3. Replication — reproducibility vs. replicability, the replication crisis, causes of replication failure, open science.
  4. Ethics — ethical frameworks, responsibilities to participants, analysis, and the scientific community.

Part 2 — Statistics

  1. Estimation — point estimates, effect sizes, maximum likelihood and Bayesian estimation.
  2. Inference — sampling variation, p-values, Bayes factors, confidence intervals, the Neyman–Pearson approach.
  3. Models — regression, GLMs, linear mixed-effects models, covariates, robustness checks.

Part 3 — Planning

  1. Measurement — reliability, validity, how to select a good measure, implicit vs. explicit behavior.
  2. Design — factorial designs, between- vs. within-participant, internal/external validity of manipulations.
  3. Sampling — probability sampling, the WEIRD problem, collider and attrition bias, sample-size planning.

Part 4 — Execution

  1. Preregistration — the garden of forking paths, researcher degrees of freedom, how to preregister.
  2. Data collection — informed consent, the research experience, piloting, data quality.
  3. Project management — organizing projects, file naming, data management, the FAIR principles.

Part 5 — Reporting

  1. Writing — clear scientific writing, reproducible reports, responsible authorship and citation.
  2. Visualization — show the design, facilitate comparison, show the data, exploratory visualization.
  3. Meta-analysis — evidence synthesis, fixed- vs. random-effects, bias in meta-analysis.
  4. Conclusion — summarizing the approach and pushing the field forward.

How to use this section

  • As a reference: jump to any chapter from the list above. Each chapter stands alone and links back here.
  • As a path: read Part 1 → Part 5 in order to follow an experiment from idea to published, reproducible result.
  • As a bridge: each chapter ends with "Connections to Kuakua" linking to related entries in Research Methods and Psychology Research (e.g., the Stroop effect illustrates design and measurement in practice).

Connections to the rest of Kuakua

Experimentology is the methodology backbone for Kuakua's experiment content. Start here when you want the "why" behind a paradigm:

Last updated: 2026-10-20

Menu
Loading