Introduction to stochastic processes
program LNMB and MasterMath , Fall 2009

[Announcements]  [General information]   [Course outline]  [Archive]  [Some further reading] 

 

Announcements

11.09.2009 The sheets for the classes 1,2,5,6 have been updated (see the links below)

02.09.2009 Dear students, welcome to the course Introduction to Stochastic Processes!

Back to the top

General information

Meetings: 10.15-12.00 and 13.00-14.45 (September 7, 8, 14 and 15)
Location:Mathematical Building, Room 611AB Budapestlaan 6 Utrecht
Instructors: Nelly Litvak and Werner Scheinhardt
Course description at MasterMath
Examination: Written exam. September 28th 2009 in the Buys Ballot Lab. room 513 in Utrecht.
Text: Introduction to Probability Models Academic Press, 9th ed., 2007 Sheldon M. Ross (7th and 8th edition can also be used; brackets below [..] indicate 8th edition)

Back to the top

Course outline

September 7, a.m. 
Topics: 
Material: 
Exercises: 


Markov Chains: definition, Chapman-Kolmogorov equations, classification of states,  
Sections 4.1 - 4.3, not example 4.16. sheets 
Ross chapter 4: exercises 1,2,3,10,14,15

September 7, p.m.
Topics: 
Material: 
Exercises: 


Markov Chains: limiting probabilities, applications, time spent in transient states,  
Sections 4.4 (nor example 4.20,4.22), 4.5.1, 4.5.2 (to study by yourself), 4.6 sheets 
Ross chapter 4: exercises 20,25,29,30,57,63 (solutions for chapter 4)  problems 10 and 13 from the list (solutions)

September 8, a.m. 
Topics: 
Material: 
Exercises: 


Generating Functions and Laplace-Stieltjes Transforms; branching processes; exponential distribution 
Handouts from book of Cooper; Ross sections 4.7 and 5.2. 
Cooper, exercise 4 (page 30), exercises 4,5 (page 199);  Ross, chapter 4: exercises 64, 66; chapter 5: exercises 3,14,18
link to the book: http://www.cse.fau.edu/%7Ebob/publications/IntroToQueueingTheory_Cooper.pdf

September 8, p.m. 
Topics: 
Material: 
Exercises: 

 

Poisson process. (Sheets September 8, classes 3-4) 
Section 5.3 until example 5.19 [5.17], but without examples 5.9 and 5.17 [without example 5.15, but with example 5.8 in new edition] 
Ross chapter 5: exercises 35, 37, 38, 40, 41, 42, 43, 60, 61, 63 (Solutions September 4, classes 3-4)

September 14, a.m. 
Topics: 
Material: 
Exercises: 


Continuous time Markov Chains: introduction, birth/death process, transition probabilities  
Sections 6.1 - 6.4 until example 6.8. 
Ross chapter 6: 1,3, 6ab, 9, 10 (but don't verify forward and backward eqns), 20ab (solutions) 

September 14, p.m. 
Topics: 
Material: 
Exercises: 


CTMC: Kolmogorov's equations, limiting distribution, uniformization (Sheets September 14, classes 5-6) 
Sections 6.4 (rest), 6.5, 6.7 
Ross chapter 6: exercises 12, 13, 15, 17, 18, 20c, 21, 22, 23 (solutions) 

September 15, a.m. 
Topics: 
Material: 
Exercises: 


Renewal theory: renewal process and limit theorems 
Section 7.1, 7.2, 7.3 [including example 7.10 in new edition] sheets 
Ross chapter 7: exercises 2,4,5,10,11,12,15,19

September 15, p.m. 
Topics: 
Material: 
Exercises: 


Renewal reward process, regeneration process 
Sections 7.4,7.5 [including examples 7.16, 7.25 in new edition] sheets 
Ross, chapter 7: exercises 21,22,26,32,37,41,44,45,51 (solutions for chapter 7) 



Back to the top

Archive

Exam 2006 (Solutions)
Exam 2005 (Solutions)

Back to the top

Some further reading

Books

  • Sheldon M. Ross. Introduction to probability models. Academic Press, New York, 2003
    The course is based on this book. The book also contains most of the homework problems.
  • Rick Durrett. Essentials of stochastic processes. Springer 1999
    Elegant rigorous treatment of most of the topics
  • James R. Norris. Markov Chains. Cambridge University Press, 1997.
    Classical reference on Markov chains
  • David Aldous and James A. Fill. Reversible Markov Chains and Random Walks on Graphs. Available here .
    Advanced book on Markov chains. Highly recommended, in particular, if you've got a feeling that this all is too easy!
  • Sheldon M. Ross. Stochastic processes. Wiley, New York, 1996..
    Very good book on stochastic processes, covering most of the material, slightly more advanced than `Introduction to Probability Models'
  • William J. Stewart (1994) An Introduction to the Numerical Solution of Markov Chains. Princeton University Press
    The classical reference if you need to find a stationary distribution of a large Markov Chain by using numerical methods

Articles

  • Gely P. Basharin, Amy N. Langville, and Valeriy A. Naumov (2004) The Life and Work of A. A. Markov. Linear Algebra and its Applications. Vol. 386:3-26 pdf
    Very well written biography of A.A. Markov including some results from his work on chains. Enjoyable and useful reading!
  • Sergey Brin and Lawrence Page (1998) The anatomy of a large-scale hypertextual web search engine. In The Seventh International World Wide Web Conference WWW1998 pdf
    In this paper Google is presented for the first time to the Computer Science community. The PageRank concept is introduced as well. The paper has got 1689 citations!
  • Amy N. Langville and Carl D. Meyer (2005) Deeper Inside PageRank. Internet Mathematics Vol. 1(3):335-380 pdf
    A solid and well written survey on PageRank. Also useful if you want to know more about the linear algebra framework and numerical methods for Markov chains


Back to the top