You are here

The Uncertainty Analysis of Model Results

Eduard Hofer
Publication Date: 
Number of Pages: 
[Reviewed by
William J. Satzer
, on

Computer models are everywhere. Where once they were found mostly in scientific and engineering environments, they have now proliferated and invaded virtually all aspects of our lives. Some of these are enormous — like models of global climate and the world economy — while others are small and special-purpose with a very specific focus, such as estimating the expected manufacturing cost of a single product.

Unfortunately, a common reaction to computer models is uncritical acceptance. Of course, there are cases (like the global climate model) where the results of the modeling are the subject of furious contention. But even when disagreements arise, they most often focus on outcomes and ignore the complicated innards of the models. This isn’t surprising, since many models are implemented with thousands (or tens of thousands) of lines of computer code and huge numbers of parameters and input variables.

In this book the author describes an approach to uncertainty analysis for computer models. He begins with an imagined dialog in a business conference intended to review a computer model. The hypothetical model he describes was designed to compare two design options to compare their compliance with a safety limit involving a critical temperature. The modeler is asked to justify the assumptions he made and to assess the uncertainty of his conclusions, but he has only fragmentary knowledge of the uncertainty of his results and so cannot justify his conclusions. The author suggests that the methodology he proposes would have avoided the embarrassment — and possibly also a serious safety problem.

It is important to note that the questions dealt with here are not about model validation in the sense of comparing and evaluating differences between model output and actual data. Instead the author is considering the analysis of predictive models before any actual data is available.

Model uncertainty analysis, when it is done now, often relies on exploring the effect of variations of parameters. One especially common technique is to assign each parameter in the model a probability distribution (with its own associated parameters) and then to use Monte Carlo techniques to identify the distribution of possible model outcomes. This, of course, introduces other uncertainties (often unacknowledged) of how the various probability distributions and their parameters are selected.

The author clearly sees that we need a more complete and more disciplined approach. He notes that model uncertainty has two primary components; he calls them epistemic and aleatoric. Epistemic uncertainty refers to things that we just don’t know and can possibly learn, and aleatoric means components that have some aspects of randomness. Most of the author’s methodology deals with the epistemic aspects.

The book is not an easy read. Long lists of items appear frequently throughout the book. The writing is often wordy and uses an idiosyncratic terminology that is difficult to pin down. The book also has no index. The best parts of the book are the two practical examples at the end that are sufficiently concrete that one can attach clear meaning to the various steps. One of these carries out the author’s full uncertainty analysis for a population dynamics model of the Peruvian anchovy. The other does the same for a model that attempts to reconstruct the doses people received from accidental exposure to a carcinogen.

The author is attacking a very difficult question and his method at least identifies an approach and offers some examples of how it works. A lingering question is how practical his approach is for complex models.

Bill Satzer ([email protected]) was a senior intellectual property scientist at 3M Company. His training is in dynamical systems and particularly celestial mechanics; his current interests are broadly in applied mathematics and the teaching of mathematics.

See the table of contents in the publisher's webpage.