Understanding Public Opinion Polls

On November 9, 2016, millions of Americans woke up to discover, to their astonishment, that Donald Trump, rather than Hillary Clinton, was to be the next President of the United States. One reason they were astonished is because virtually every public opinion poll had forecast exactly the opposite result. Quite a lot of ink has been spilled since then attempting to explain the reasons for this massive error. As an illustration, a google search of the words (without quotation marks) “election polls 2016 what went wrong” produces literally millions of links.

Of course, this is not the first time in American history that polls and surveys have turned out to be spectacularly wrong. Many people of my generation, for example, are very familiar with the classic 1948 photograph of President Truman gleefully holding up a copy of the Chicago Daily Tribune containing the headline “Dewey Defeats Truman”. And the cover of the November 14, 1936 issue of The Literary Digest consisted of a large circle with the phrase “Is Our Face Red!” contained within it, after that magazine incorrectly predicted that Alf Landon would handily defeat Franklin Roosevelt in the presidential election held a few weeks earlier. (The magazine folded shortly after this.)

Stories like this tell us that creating, implementing and interpreting a good public opinion poll is not easy, and that observation, in turn, brings us to the book now under review. Its title notwithstanding, this is more than just a book about understanding polls; it is also about designing, analyzing and criticizing them. The book is designed for people “who are in some way confronted with polls”, and is intended to help them “establish whether a poll is good or bad, and whether they safely can use the poll results, or whether it is better to ignore the poll.” The book is also intended as “a useful source of information for those who want to carry out a poll themselves.”

Of course, in this day and age, most everybody falls into the category of people who are at some time “confronted with polls”, and so there is considerable value in helping to instill some sense of “poll literacy” among the general population, not just people whose livelihoods involve poll-reading.

The author proposes nine questions about a public opinion poll which, if all are answered in the affirmative, suggest that the poll is a good one:

• Is there a research report explaining how the poll was set up and carried out?
• Is the poll commissioned or sponsored by an organization that has no interest in its outcomes?
• Is the target population of the poll clearly defined?
• Is a copy of the questionnaire included in the research report, or otherwise available?
• Is the sample a random sample for which each person in the target population has a positive probability of selection?
• Are the initial (gross) sample size and the realized (net) sample size (number of respondents?) reported?
• Is the response rate sufficiently high, say higher than 50%?
• Have the outcomes been corrected (by adjustment weighting) for selective nonresponse?
• Are the margins of error specified?

Much of the book can be viewed as an elaboration on these questions. There are entire chapters, for example, on sampling, estimation and response rate. In addition to discussing issues raised by these questions, the author also includes a chapter on the history of polling, as well as chapters on particular issues raised by online polling and election polls.

The book has been deliberately written so as to be broadly accessible to a lay audience. As a result, no background in mathematics or statistics is assumed, and very few technicalities are discussed. An occasional formula does make its way into the book, but nothing much is done with them. It’s too bad, I think, that the author didn’t add some extra chapters or appendices for the more sophisticated reader, perhaps covering such topics as the use of conditional probability in polls. (For example, when dealing with sensitive questions (e.g., “have you ever cheated on an exam in college?”) one could use randomized responses — have a student flip a coin to determine whether to lie or not—and then use conditional probability to approximate the percentage of students who actually have cheated.)

The author’s writing style is quite clear, with lots of examples given of actual polls (both from the United States and abroad). All three of the examples listed in the first paragraphs of this review are discussed in the text, for example. (With regard to the Presidential election polls, the author lists four possible causes for the problem, and states that the American Association for Public Opinion Research will be investigating the issue further.)

Because of the clarity of writing, use of specific examples and avoidance of technical details, this book should be easily comprehensible by its intended audience of people who use or design polls. Perhaps, on occasion, it can be a shade too easy to read; some passages struck me as rather simplistic. (Example: “There are many polls, particularly during election campaigns. Not all polls are good. There are many bad polls.”)

A few quibbles: first, there are a large number of typos (“level of educational”, “an import means of communication”, “an overview of the every changing landscape”, “second halve of the twentieth century”) and examples of bad English (“roll a dice”, “became quickly very popular”, “it should be clear how large the difference at most can be”) throughout the text. Typos in a book are not uncommon, but here they are sufficiently frequent to be distracting.

Another quibble concerns the lack of any bibliography. This is particularly vexing here, because the book is replete with specific references to articles or books, as in “Krosnick (1991) and Tourangeau et al. (2004) express concern about…”. A reader who would like to follow up on this, however, will find neither any further identification of these references, nor any hint as to where they can be found; it’s not even clear, for example, whether these references are to books, chapters in books, or journal articles. One can, I suppose, search for them online (just for fun, I did find the Krosnick article without any trouble) but not everybody reads a book with a computer close at hand and, in any event, I do not believe it is a reader’s responsibility to search out the identity of partial references given by an author.

Despite these issues, this book should serve its intended purpose nicely: people who work with polls or who need to design them will find this a useful, practical, down-to-earth primer on that subject. In addition, while this is not intended as a textbook (there are no exercises, for example), I think that some instructors might find it useful as a handy reference. For example, people who teach “mathematics literacy” courses, particularly those that touch on social choice and politics, could possibly get some useful lecture ideas from this book.

Mark Hunacek (mhunacek@iastate.edu) teaches mathematics at Iowa State University.