Decision Processes in Dynamic Probabilistic Systems

1 Semi-Markov and Markov Chains.- 1.1 Definitions and basic properties.- 1.1.1 Discrete time semi-Markov and Markov behaviour of systems.- 1.1.2 Multi-step transition probability.- 1.1.3 Semi-Markov processes.- 1.1.4 State occupancy and waiting-time statistics.- 1.1.5 Non-homogeneous Markov processes.- 1.1.6 The limit theorem.- 1.1.7 Effect of small deviations in the transition probability matrix.- 1.1.7.1 Limits of some important characteristics for Markov chains.- 1.2 Algebraic and analytical methods in the study of Markovian systems.- 1.2.1 Eigenvalues and eigenvectors.- 1.2.2 Stochastic matrices.- 1.2.3 Perron-Frobenius theorem.- 1.2.4 The Geometric transformation (the z-transform).- 1.2.5 Exponential transformation (Laplace transform).- 1.3 Transient and recurrent processes.- 1.3.1 Transient processes.- 1.3.2 The study of recurrent state occupancy in Markov processes.- 1.4 Markovian populations.- 1.4.1 Vectorial processes with a Markovian structure.- 1.4.2 General branching processes.- 1.5 Partially observable Markov chains.- 1.5.1 The core process.- 1.5.2 The observation process.- 1.5.3 The state of knowledge and its dynamics.- 1.5.4 Examples.- 1.6 Rewards and discounting.- 1.6.1 Rewards for sequential decision processes.- 1.6.2 Rewards in decision processes with Markov structure.- 1.6.3 Markovian decision processes with and without discounting.- 1.7 Models and applications.- 1.7.1 Real systems with a Markovian structure.- 1.7.2 Model formulation and practical results.- 1.7.2.1 A semi-Markov model for hospital planning.- 1.7.2.2 System reliability.- 1.7.2.3 A Markovian interpretation for PERT networks.- 1.8 Dynamic-decision models for clinical diagnosis.- 1.8.1 Pattern recognition.- 1.8.2 Model optimization.- 2 Dynamic and Linear Programming.- 2.1 Discrete dynamic programming.- 2.2 A linear programming formulation and an algorithm for computation.- 2.2.1 A general formulation for the LP problem and the Simplex method.- 2.2.2 Linear programming - a matrix formulation.- 3 Utility Functions and Decisions under Risk.- 3.1 Informational lotteries and axioms for utility functions.- 3.2 Exponential utility functions.- 3.3 Decisions under risk and uncertainty; event trees.- 3.4 Probability encoding.- 4 Markovian Decision Processes (Semi-Markov and Markov) with Complete Information (Completely Observable).- 4.1 Value iteration algorithm (the finite horizon case).- 4.1.1. Semi-Markov decision processes.- 4.1.2 Markov decision processes.- 4.2 Policy iteration algorithm (the finite horizon optimization).- 4.2.1 Semi-Markov decision processes.- 4.2.2 Markov decision processes.- 4.3 Policy iteration with discounting.- 4.3.1 Semi-Markov decision processes.- 4.3.2 Markov decision processes.- 4.4 Optimization algorithm using linear programming.- 4.4.1 Semi-Markov decision process.- 4.4.2 Markov decision processes.- 4.5 Risk-sensitive decision processes.- 4.5.1 Risk-sensitive finite horizon Markov decision processes.- 4.5.2 Risk-sensitive infinite horizon Markov decision processes.- 4.5.3 Risk-sensitive finite horizon semi-Markov decision processes.- 4.5.4 Risk-sensitive infinite horizon semi-Markov decision processes.- 4.6 On eliminating sub-optimal decision alternatives in Markov and semi-Markov decision processes.- 4.6.1 Markov decision processes.- 4.6.2 Semi-Markov decision processes with finite horizon.- 5 Partially Observable Markovian Decision Processes.- 5.1 Finite horizon partially observable Markov decision processes.- 5.2 The infinite horizon with discounting for partially observable Markov decision processes.- 5.2.1 Model formulation.- 5.2.2 The concept of finitely transient policies.- 5.2.3 The function C(? ?) approximated as a Markov process with a finite number of states.- 5.3 A useful policy iteration algorithm, for discounted (? 2.- 5.4 The infinite horizon without discounting for partially observable Markov processes.- 5.4.1 Model formulations.- 5.4.2 Cost of a stationary policy.- 5.4.3 Policy improvement phase.- 5.4