Howard Wainer is the author of several instructive and entertaining books on the practice of statistics and especially on the use of graphics and visualization. These include Graphic Discovery: A Trout in the Milk and Other Visual Adventures and Visual Revelations: Graphical Tales of Fate and Deception from Napoleon Bonaparte to Ross Perot. Wainer has also published widely on technical issues in testing and psychometrics. Those of his previous books aimed at broader audiences are witty and generally easy-going. In the current book, however, Wainer is pretty riled up. Consider these words from the preface:
I don’t know whether it is the age we live in, or the age I have lived to, but whichever, I have lately found myself shouting at the TV screen disturbingly often. … [S]ome of the blame for these untoward outbreaks can be traced directly to the remarkable dopiness that substitutes for wisdom in modern society.
Wainer mainly considers the use of tests and test scores in support of educational decision making. Much of his frustration is aimed at those who make important educational decisions based on anecdotes (or spurious data) instead of evidence. Wainer worked at the Educational Testing Service (ETS) for more than twenty years, and then moved to the National Board of Medical Examiners. He knows testing, and he is intimately familiar with its strengths and weaknesses. He has convincing arguments, even for those of us who are skeptical about the large scale use of standardized testing.
In 2008 the National Association for College Admission Counseling made several recommendations. These included making the SAT or ACT exams optional, replacing them with achievement tests, and eliminating a rigid cut off score in the PSAT exam as a qualification for National Merit scholarships. Wainer argues that these are all ill-considered. All the available evidence shows that the aptitude exams are the best predictors of success in college. As for replacing them with achievement exams, it is difficult or impossible to compare, for example, the performance on one student on a French achievement exam and another’s in Chemistry, at least not without having related data from aptitude exams.
Several of the chapters delve more deeply into psychometric questions. One significant issue is how to compare performance across groups of individuals who took different forms of a test. This is a subtle question without a clear solution, and it gets harder as the differences between the tests increase. Even versions of (nominally) the same test in English and Spanish can make fair comparison of performance difficult.
Two topics here might be of special interest to readers. The first is the already mentioned question of optional SATs for college admission. Wainer provides data from one year at Bowdoin College (where SATs are optional) that frame the issue, and then broadens the analysis by adding data from five other schools that do require either the SAT or the ACT exams, and have mean scores comparable to Bowdoin’s. (Wainer had access to a SAT scores from all first year students at Bowdoin, including scores from students who chose not to submit them.) The correlations of scores with first year grade point averages are remarkably consistent across schools.
The second topic of special interest is the effect of allowing students to choose which exam questions they respond to; this occurs, for example, in an advanced placement exams when a student has a choice of essay topics. It is distinctly counter-intuitive, but choice serves many students — and some distinct groups — very poorly. On average, students of lesser ability tend to choose less wisely than those who perform better. Indeed, in some cases, one can virtually predict the score knowing only the choice the student made.
I would recommend this book to anyone with an interest in testing, especially for college admissions or advanced placement. Not all the chapters will be of interest to everyone, but Wainer is a gifted writer with a notable talent for analyzing and presenting data.
Bill Satzer (email@example.com) is a senior intellectual property scientist at 3M Company, having previously been a lab manager at 3M for composites and electromagnetic materials. His training is in dynamical systems and particularly celestial mechanics; his current interests are broadly in applied mathematics and the teaching of mathematics.
Chapter 1: O n the Value of Entrance Exams: What Happens When the SAT Is Made Optional? 8
Chapter 2: O n Substituting Achievement Tests for Aptitude Tests in College Admissions 20
Chapter 3: O n Rigid Decision Rules for Scholarships 29
Chapter 4: The Aptitude-Achievement Connection: Using an Aptitude Test to Aid in Allocating Educational Resources 32
Chapter 5: C omparing the Incomparable: On the Importance of Big Assumptions and Scant Evidence 57
Chapter 6: O n Examinee Choice in Educational Testing 73
Chapter 7: What If Choice Is Part of the Test? 103
Chapter 8: A Little Ignorance Is a Dangerous Thing: How Statistics Rescued a Damsel in Distress 110
Chapter 9: A ssessing Teachers from Student Scores: On the Practicality of Value-Added Models 120
Chapter 10: S hopping for Colleges When What We Know Ain't 139
Chapter 11: O f CAT s and Claims: The First Step toward Wisdom 147