Hide menu

Introduction to Markov decision processes

DF17100, 2009VT

Status Archive
School National Graduate School in Computer Science (CUGS)
Division TECH-OU
Owner Lars Karlsson

  Log in  

Course plan


Two lectures/exercises; approx 3h lec + 2 h ex each.

Recommended for

Interesting for all CUGS students, but in particular relevant for those working with systems that need to interact with physical environment or other agents.

The course was last given

Fall 2004


To obtain a basic understanding of what a Markov decision process is and how it can be used.


Basic knowledge in probability theory and optimization.


The course gives an introduction to Markov decision processes, which is a tool for optimal decision making in situations where the outcomes of the decision-making system's actions have an element of randomness, MPDs are used in computer science, control, biology, economics and a number of other fields. More specifically, the course adresses:

* Markov Chains

* Markov Decision Processes

* Solving MDPs: value iteration and policy iteration

* Partially Observable Markov Decision Processes (POMDPs)

* Applications: CS (planning, learning etc), control, biology, economy, and so on.


Intensive course


Martin L. Puterman: Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley-Interscience, 1994. ISBN: 0471619779.

Lecture notes and articles.


Lars Karlsson


Lars Karlsson


Home assignment.


4 hp

Organized by

AASS, Örebro university


Schedule: prel February-April, 2009.
The course will be given in Örebro, and scheduled to suit students that need to travel.

Page responsible: Director of Graduate Studies
Last updated: 2012-05-03