Markov processes are stochastic processes used to model the random evolution of “memory-less” systems, i.e. systems where the probability of a transition to a future state depends uniquely on the present state and not on the past.
After a first chapter devoted to introduce the fundamentals on Markov chains, the author explains how a Markov process can be applied to such diverse fields as mathematical finance, the genome and the analysis of queues and networks. These chapters are completely independent from each other, so that one can browse through the book in many ways, according to one’s interests and needs.
Well-written, this book is suitable as a textbook for teaching a postgraduate course on applied Markov processes. There is a wealth of interesting exercises (solutions to selected exercises are provided at the end); several exercises throughout the book are intended to be implemented on some suitable software, for instance MATLAB. The necessary prerequisites are: linear algebra, probability theory, random variables, calculus (measure theory + ordinary and partial differential equations). Most of the necessary background can be found in Sheldon Ross’ wonderful book, Introduction to Probability Models.
With the exception of chapter 9 on the mathematical finance, all the important results are carefully proven; otherwise, precise references are given. As the author indicates in the preface, working on the exercises is essential for mastering the content of the book (especially because the examples are not so frequent in the text).
Fabio Mainardi earned a PhD in Mathematics at the University of Paris 13. His research interests are mainly Iwasawa theory, p-adic L-functions and the arithmetic of automorphic forms. At present, he works in a "classe préparatoire" in Geneva. He may be reached at firstname.lastname@example.org.
1. Simulations and the Monte Carlo method.
1.1 Description of the method.
1.2 Convergence theorems.
1.3 Simulation of random variables.
1.4 Variance reduction techniques.
2. Markov chains.
2.1 Definitions and elementary properties.
2.3 Strong Markov property.
2.4 Recurrent and transient states.
2.5 The irreducible and recurrent case.
2.6 The aperiodic case.
2.7 Reversible Markov chain.
2.8 Rate of convergence to equilibrium.
2.9 Statistics of Markov chains.
3. Stochastic algorithms.
3.1 Markov chain Monte Carlo.
3.2 Simulation of the invariant probability.
3.3 Rate of convergence towards the invariant probability.
3.4 Simulated annealing.
4. Markov chains and the genome.
4.1 Reading DNA.
4.2 The i.i.d. model.
4.3 The Markov model.
4.4 Hidden Markov models.
4.5 Hidden semi-Markov model.
4.6 Alignment of two sequences.
4.7 A multiple alignment algorithm.
5. Control and filtering of Markov chains.
5.1 Deterministic optimal control.
5.2 Control of Markov chains.
5.3 Linear quadratic optimal control.
5.4 Filtering of Markov chains.
5.5 The Kalman-Bucy filter.
5.6 Linear-quadratic control with partial observation.
6. The Poisson process.
6.1 Point processes and counting processes.
6.2 The Poisson process.
6.3 The Markov property.
6.4 Large time behaviour.
7. Jump Markov processes.
7.1 General facts.
7.2 Infinitesimal generator.
7.3 The strong Markov property.
7.4 Embedded Markov chain.
7.5 Recurrent and transient states.
7.6 The irreducible recurrent case.
7.8 Markov models of evolution and phylogeny.
7.9 Application to discretized partial differential equations.
7.10 Simulated annealing.
8. Queues and networks.
8.1 M/M/1 queue.
8.2 M/M/1/K queue.
8.3 M/M/s queue.
8.4 M/M/s/s queue.
8.5 Repair shop.
8.6 Queues in series.
8.7 M/G/1 queue.
8.8 M/G/1 queue.
8.9 Open Jackson network.
8.10 Closed Jackson network.
8.11 Telephone network.
8.12 Kelly networks.
9. Introduction to mathematical finance.
9.1 Fundamental concepts.
9.2 European options in the discrete model.
9.3 The Black-Scholes model and formula.
9.4 American options in the discrete model.
9.5 American options in the Black-Scholes model.
9.6 Interest rate and bonds.
10. Solutions to selected exercises.
10.1 Chapter 1.
10.2 Chapter 2.
10.3 Chapter 3.
10.4 Chapter 4.
10.5 Chapter 5.
10.6 Chapter 6.
10.7 Chapter 7.
10.8 Chapter 8.
10.9 Chapter 9.