You are here

An Introduction to Statistical Computing: A Simulation-based Approach

Jochen Voss
Publication Date: 
Number of Pages: 
Wiley Series in Computational Statistics
[Reviewed by
William J. Satzer
, on

Statistical computing in its broadest sense is an ever-growing field far too extensive to be covered in a single text. The current book has a far more manageable scope, notwithstanding its title. Its focus is on the use of Monte Carlo methods to simulate random systems and explore statistical models.

The author proposes to investigate three different kinds of questions about systems with random components. The first is to understand the normal or nominal behavior of the system. The second is to quantify the variability and estimate the magnitude of random fluctuations. The third is to explore exceptional behavior and attempt to determine the probability of specific untypical events.

The book begins with a strong section on pseudo-random number generation. This includes a discussion of methods of generating uniformly distributed pseudo-random numbers, and of using those to generate random variables with other desired distributions. This is a critical piece for getting reliable Monte Carlo simulation results. (I have run such simulations with more than a hundred random variables to generate hundreds of realizations of a physical process. A faulty pseudo-random number generator in a situation like this can introduce subtle but unreal correlations between random variables and cast doubt on all the simulation results.)

Having described how to get the pseudo-random numbers need for statistical simulations, the author proceeds to discuss how appropriate correlations can be introduced between them to get, for example, a desired covariance matrix. With all the tools in place, we move on to the heart of the book: Monte Carlo methods and Markov Chain Monte Carlo methods. The material here is pretty standard but the treatment is a little unusual. Books that focus on simulation generally explain what to do with a minimum of mathematical background. Mathematical treatments usually don’t include algorithms and code. This book has a mix of both.

Beyond the various Monte Carlo techniques, the author also has a short section on methods that can be used when no useful statistical model is available. These include Approximate Bayesian Computation (ABC) and resampling methods (primarily the Bootstrap). Finally, continuous time models get a chapter of their own that includes a discussion of Brownian motion and a bit about stochastic differential equations.

While there are a variety of examples — dealing, for example, with image analysis and the Ising model — there are not enough. The chapters on Monte Carlo techniques would benefit from at least a couple of extended examples in real applications. Several more small-scale examples would also be very desirable.

It is not clear where this book might fit in the curriculum. The author aimed it at an undergraduate audience and has limited the mathematical background accordingly, but the topics are rather specialized. This could be a supplementary source for a course in simulation and modeling, especially where Monte Carlo techniques are included.

Bill Satzer ( is a senior intellectual property scientist at 3M Company, having previously been a lab manager at 3M for composites and electromagnetic materials. His training is in dynamical systems and particularly celestial mechanics; his current interests are broadly in applied mathematics and the teaching of mathematics.

List of algorithms ix

Preface xi

Nomenclature xiii

1 Random number generation 1

1.1 Pseudo random number generators 2

1.1.1 The linear congruential generator 2

1.1.2 Quality of pseudo random number generators 4

1.1.3 Pseudo random number generators in practice 8

1.2 Discrete distributions 8

1.3 The inverse transform method 11

1.4 Rejection sampling 15

1.4.1 Basic rejection sampling 15

1.4.2 Envelope rejection sampling 18

1.4.3 Conditional distributions 22

1.4.4 Geometric interpretation 26

1.5 Transformation of random variables 30

1.6 Special-purpose methods 36

1.7 Summary and further reading 36

Exercises 37

2 Simulating statistical models 41

2.1 Multivariate normal distributions 41

2.2 Hierarchical models 45

2.3 Markov chains 50

2.3.1 Discrete state space 51

2.3.2 Continuous state space 56

2.4 Poisson processes 58

2.5 Summary and further reading 67

Exercises 67

3 Monte Carlo methods 69

3.1 Studying models via simulation 69

3.2 Monte Carlo estimates 74

3.2.1 Computing Monte Carlo estimates 75

3.2.2 Monte Carlo error 76

3.2.3 Choice of sample size 80

3.2.4 Refined error bounds 82

3.3 Variance reduction methods 84

3.3.1 Importance sampling 84

3.3.2 Antithetic variables 88

3.3.3 Control variates 93

3.4 Applications to statistical inference 96

3.4.1 Point estimators 97

3.4.2 Confidence intervals 100

3.4.3 Hypothesis tests 103

3.5 Summary and further reading 106

Exercises 106

4 Markov Chain Monte Carlo methods 109

4.1 The Metropolis–Hastings method 110

4.1.1 Continuous state space 110

4.1.2 Discrete state space 113

4.1.3 Random walk Metropolis sampling 116

4.1.4 The independence sampler 119

4.1.5 Metropolis–Hastings with different move types 120

4.2 Convergence of Markov Chain Monte Carlo methods 125

4.2.1 Theoretical results 125

4.2.2 Practical considerations 129

4.3 Applications to Bayesian inference 137

4.4 The Gibbs sampler 141

4.4.1 Description of the method 141

4.4.2 Application to parameter estimation 146

4.4.3 Applications to image processing 151

4.5 Reversible Jump Markov Chain Monte Carlo 158

4.5.1 Description of the method 160

4.5.2 Bayesian inference for mixture distributions 171

4.6 Summary and further reading 178

4.6 Exercises 178

5 Beyond Monte Carlo 181

5.1 Approximate Bayesian Computation 181

5.1.1 Basic Approximate Bayesian Computation 182

5.1.2 Approximate Bayesian Computation with regression 188

5.2 Resampling methods 192

5.2.1 Bootstrap estimates 192

5.2.2 Applications to statistical inference 197

5.3 Summary and further reading 209

Exercises 209

6 Continuous-time models 213

6.1 Time discretisation 213

6.2 Brownian motion 214

6.2.1 Properties 216

6.2.2 Direct simulation 217

6.2.3 Interpolation and Brownian bridges 218

6.3 Geometric Brownian motion 221

6.4 Stochastic differential equations 224

6.4.1 Introduction 224

6.4.2 Stochastic analysis 226

6.4.3 Discretisation schemes 231

6.4.4 Discretisation error 236

6.5 Monte Carlo estimates 243

6.5.1 Basic Monte Carlo 243

6.5.2 Variance reduction methods 247

6.5.3 Multilevel Monte Carlo estimates 250

6.6 Application to option pricing 255

6.7 Summary and further reading 259

Exercises 260

Appendix A Probability reminders 263

A.1 Events and probability 263

A.2 Conditional probability 266

A.3 Expectation 268

A.4 Limit theorems 269

A.5 Further reading 270

Appendix B Programming in R 271

B.1 General advice 271

B.2 R as a Calculator 272

B.2.1 Mathematical operations 273

B.2.2 Variables 273

B.2.3 Data types 275

B.3 Programming principles 282

B.3.1 Don’t repeat yourself! 283

B.3.2 Divide and conquer! 286

B.3.3 Test your code! 290

B.4 Random number generation 292

B.5 Summary and further reading 294

Exercises 294

Appendix C Answers to the exercises 299

C.1 Answers for Chapter 1 299

C.2 Answers for Chapter 2 315

C.3 Answers for Chapter 3 319

C.4 Answers for Chapter 4 328

C.5 Answers for Chapter 5 342

C.6 Answers for Chapter 6 350

C.7 Answers for Appendix B 366

References 375

Index 379