Randomization Tests

 STATISTICAL TESTS THAT DO NOT REQUIRE RANDOM SAMPLING
Randomization Tests
Numerical Examples
Randomization Tests and Nonrandom Samples
The Prevalence of Nonrandom Samples in Experiments
The Irrelevance of Random Samples for the Typical Experiment
Generalizing from Nonrandom Samples
Intelligibility
Respect for the Validity of Randomization Tests
Versatility
Practicality
Precursors of Randomization Tests
Other Applications of Permutation Tests
Questions and Exercises
Notes
References

RANDOMIZED EXPERIMENTS
Unique Benefits of Experiments
Experimentation without Manipulation of Treatments
Matching: A Precursor of Randomization
Randomization of Experimental Units
Experimental Units
Groups as Experimental Units
Control over Confounding Variables
Between-Subject and Within-Subject Randomization
Conventional Randomization Procedures
Randomization Procedures for Randomization Tests
Further Reading
Questions and Exercises

CALCULATING P-VALUES
Introduction
Systematic Reference Sets
Criteria of Validity for Randomization Tests
Randomization Test Null Hypotheses
Permuting Data for Experiments with Equal Sample Sizes
Monte Carlo Randomization Tests
Equivalent Test Statistics
Randomization Test Computer Programs
Writing Programs for Randomization Tests
How to Test Systematic Data Permutation Programs
How to Test Random Data Permutation Programs
Nonexperimental Applications of the Programs
Questions and Exercises
References

BETWEEN-SUBJECTS DESIGNS
Introduction
One-Way ANOVA with Systematic Reference Sets
A Simpler Test Statistic Equivalent to F
One-Way ANOVA with Equal Sample Sizes
One-Way ANOVA with Random Reference Sets
Analysis of Covariance
One-Tailed t Tests and Predicted Direction of Difference
Simpler Equivalent Test Statistics to t
Tests of One-Tailed Null Hypotheses for t Tests
Unequal-N One-Tailed Null Hypotheses
Fast Alternatives to Systematic Data Permutation for Independent t Tests
Independent t Test with Random Reference Sets
Independent t Test and Planned Comparisons
Independent t Test and Multiple Comparisons
Loss of Experimental Subjects
Ranked Data
Dichotomous Data
Outliers
Questions and Exercises
References

FACTORIAL DESIGNS
Advantages of Randomization Tests for Factorial Designs
Factorial Designs for Completely Randomized Experiments
Proportional Cell Frequencies
Program for Tests of Main Effects
Completely Randomized Two-Factor Experiments
Completely Randomized Three-Factor Experiments
Interactions in Completely Randomized Experiments
Randomization Test Null Hypotheses and Test Statistics
Designs with Factor-Specific Dependent Variables
Dichotomous and Ranked Data
Fractional Factorial and Response Surface Designs
Questions and Exercises
References

REPEATED-MEASURES AND RANDOMIZED BLOCK DESIGNS
Carry-Over Effects in Repeated-Measures Designs
The Power of Repeated-Measures Tests
Systematic Listing of Data Permutations
A Nonredundant Listing Procedure
St2 as an Equivalent Test Statistic to F
Repeated-Measures ANOVA with Systematic Data Permutation
Repeated-Measures ANOVA with Random Data Permutation
Correlated t Test with Systematic Data Permutation
Fast Alternatives to Systematic Data Permutation for Correlated t Tests
Correlated t Test with Random Data Permutation
Correlated t Test and Planned Comparisons
Correlated t Test and Multiple Comparisons
Rank Tests
Dichotomous Data
Counterbalanced Designs
Outliers
Factorial Experiments with Repeated Measures
Interactions in Repeated-Measures Experiments
Randomized Block Designs
Randomized Complete Blocks
Incomplete Blocks
Treatments-by-Subjects Designs
Disproportional Cell Frequencies
Test Statistic for Disproportional Cell Frequencies
Data Adjustment for Disproportional Cell Frequency Designs
Restricted-Alternatives Random Assignment
Combining P-Values
Additive Method of Combining P-Values
Combining One-Tailed and Two-Tailed P-Values
Questions and Exercises
References

MULTIVARIATE DESIGNS
Importance of Parametric Assumptions Underlying MANOVA
Randomization Tests for Conventional MANOVA
Custom-Made Multivariate Randomization Tests
Effect of Units of Measurement
Multivariate Tests Based on Composite z Scores
Combining t or F Values over Dependent Variables
A Geometrical Model
Tests of Differences in Composition
Evaluation of Three MANOVA Tests
Multivariate Factorial Designs
Combining Univariate and Multivariate P-Values
Questions and Exercises
References

CORRELATION
Determining P-Values by Data Permutation
Computer Program for Systematic Data Permutation
Correlation with Random Data Permutation
Multivariate Correlation
Point-Biserial Correlation
Correlation between Dichotomous Variables
Spearman's Rank Correlation Procedure
Kendall's Rank Correlation Procedure
Questions and Exercises
References

TREND TESTS
Goodness-of-Fit Trend Test
Power of the Goodness-of-Fit Trend Test
Test Statistic for the Goodness-of-Fit Trend Test
Computation of Trend Means
Computer Program for Goodness-of-Fit Trend Test
Modification of the Goodness-of-Fit Trend Test Statistic
Correlation Trend Test
Correlation Trend Test for Factorial Designs
Disproportional Cell Frequencies
Data Adjustment for Disproportional Cell Frequency Designs
Combining of P-Values for Trend Tests for Factorial Experiments
Repeated-Measures Trend Tests
Differences in Trends
Correlation Trend Test and Simple Correlation
Ordered Levels of Treatments
Ranked and Dichotomous Data
Questions and Exercises
References

MATCHING AND PROXIMITY EXPERIMENTS
Randomization Tests for Matching
Randomization Tests of Proximity
Matching and Proximity Tests Based on Random Selection of Treatment Levels
Questions and Exercises
References

N-OF-1 DESIGNS
The Importance of N-of-1 Designs
Fisher's Lady-Tasting-Tea Experiment
The Concept of Choosing as a Random Process
Limitations of the Random Sampling Model for N-of-1 Experiments
Random Assignment Model
Carry-Over Effects
The N-of-1 Randomization Test: An Early Model
Factorial Experiments
Randomized Blocks
Correlation
Operant Research and Treatment Blocks
ABAB Design
Random Assignment of Treatment Blocks to Treatments
Randomization Tests for Treatment Intervention
Effects of Trends
Randomization Tests for Intervention and Withdrawal
Multiple Schedule Experiments
Power of N-of-1 Randomization Tests
Replicated N-of-1Experiments
N-of-1 Clinical Trial Facilities
Single-Cell and Other Single-Unit Neuroscience Experiments
Books on N-of-1 Design and Analysis
Software for N-of-1 Randomization Tests
Questions and Exercises
References

TESTS OF QUANTITATIVE LAWS
Generic and Specific Null Hypotheses
The Referent of a Law or Model
Test of Incremental Effects
Weber's Law
Other Psychophysical Laws
Foraging Behavior of Hawks
Complications
Questions and Exercises
References

TESTS OF DIRECTION AND MAGNITUDE OF EFFECT
Tests of One-Tailed Null Hypotheses for Correlated t Tests
Other Tests of One-Tailed Null Hypotheses Using ta or (A -B) as Test Statistics
Tests of One-Tailed Null Hypotheses about Differences in Variability
Tests of One-Tailed Null Hypotheses for Correlation
Testing Null Hypotheses about Magnitude of Effect
Testing Null Hypotheses about Specific Additive Effects
Questions and Exercises
References

FUNDAMENTALS OF VALIDITY
Randomization Tests as Distribution-Free Tests
Differences between Randomization Test Theory and Permutation Test Theory
Parametric Tests as Approximations to Randomization Tests
Randomization Test Theory
Systematically Closed Reference Sets Permutation Groups
Data-Permuting and Randomization-Referral Procedures
Invariance of Measurements under the Null Hypothesis
General and Restricted Null Hypotheses
Reference Sets for General Null Hypotheses
Reference Subsets for General Null Hypotheses
Reference Subsets for Restricted Null Hypotheses
Reference Subsets for Planned and Multiple Comparisons
Reference Subsets for Factorial Designs
Open Reference Sets: Treatment Intervention and Withdrawal
Closed Reference Sets: Dropouts
Open Reference Sets: Permuting Residuals
Sampling a List of Randomizations
Random Data Permutation: Hypothesis Testing vs. Estimation
Stochastic Closure When Assignments Are Equally Probable
Systematic Expansion of a Random Reference Set
Random Ordering of Measurements within Treatments
Fixed, Mixed, and Random Models
Deriving One-Tailed P-Values from Two-Tailed P-Values with Unequal N
Test Statistics and Adaptive Tests
Stochastic Closure When Assignments Are Not Equally Probable
Questions and Exercises
References

GENERAL GUIDELINES AND SOFTWARE AVAILABILITY
Randomization: Multistage Model
Permuting Data: Data-Exchanging Model
Maximizing Power
Randomization Test Computer Programs on the CD
Other Computer Programs
References