# 732A46 Bayesian Learning

### Course information

#### Aims

The course aims to give a solid introduction to the Bayesian approach to statistical inference, with a view towards applications in data mining and machine learning. After an introduction to the subjective probability concept that underlies Bayesian inference, the course moves on to the mathematics of the prior-to-posterior updating in basic statistical models, such as the Bernoulli, normal and multinomial models. Linear regression and spline regression are also analyzed using a Bayesian approach. The course subsequently shows how complex models can be analyzed with simulation methods like Markov Chain Monte Carlo (MCMC). Bayesian prediction and marginalization of nuisance parameters is explained, and introductions to Bayesian model selection and Bayesian decision theory are also given.

#### Contents

- Introduction to subjective probability and the basic ideas behind Bayesian inference
- Prior-to-posterior updating in basic statistical models, such as the Bernoulli, normal and multinomial models.
- Bayesian analysis of linear and nonlinear regression models
- Shrinkage, variable selection and other regularization priors
- Bayesian analysis of more complex models with simulation methods, e.g. Markov Chain Monte Carlo (MCMC).
- Bayesian prediction and marginalization of nuisance parameters
- Introduction to Bayesian model selection
- Introduction to Bayesian decision theory.

#### Intended audience and admission requirements

This course is given primarily for students on the Master's programme

*Statistics and Data Mining*. It is also offered to Master students in other subjects and to interested Ph.D. students (with a more advanced examination).

Students admitted to the Master's programme in Statistics and Data Mining fulfill the admission requirements for the course.

Students not admitted to the Master's programme in Statistics and Data Mining should have passed:

- an intermediate course in probability and statistical inference
- a basic course in mathematical analysis
- a basic course in linear algebra
- a basic course in programming

#### Course plan

The TimeEdit schedule for the course is available here.

The labs should be done in pairs of students whereas the project is individual work.

All labs and the project should be submitted as PDFs through LISAM.

#### Module 1 - The Bayesics

**Lecture 1**: Basics concepts. Likelihood. The Bernoulli model. The Gaussian model.

**Read**: BDA Ch. 1, 2.1-2.5 | Slides

**Code**: Beta density | Bernoulli model | One-parameter Gaussian model

**Lecture 2**: Conjugate priors. The Poisson model. Prior elicitation. Noninformative priors.

**Read**: BDA Ch. 2.6-2.9 | Slides

**Lecture 3**: Multi-parameter models. Marginalization. Multinomial model. Multivariate normal model.

**Read**: BDA Ch. 3. | Slides

**Code**: Two-parameter Gaussian model | Prediction with two-parameter Gaussian model | Multinomial model

**Lab 1**: Exploring posterior distributions in one-parameter models by simulation and direct numerical evaluation.

Lab 1

#### Module 2 - Bayesian Regression and Classification

**Lecture 4**: Prediction. Making Decisions.

**Read**: BDA Ch. 9.1-9.2. | Slides

**Lecture 5**: Linear Regression. Nonlinear regression. Regularization priors.

**Read**: BDA Ch. 14 and Ch. 20.1-20.2 | Slides

**Lecture 6**: Classification. Posterior approximation. Logistic regression. Naive Bayes.

**Read**: BDA Ch. 16.1-16.3 | Slides

**Code**: Logistic and Probit Regression

**Lab 2**: Multinomial-Dirichlet and Polynomial regression.

Lab 2 | Japanese temperature data.

#### Module 3 - More Advanced Models and MCMC Simulation

**Lecture 7**: Bayesian computations. Monte Carlo integration. Non-Markovian simulation.

**Read**: BDA Ch. 10-11 | Slides

**Code**: Gibbs sampling for a bivariate normal | Gibbs sampling for a mixture of normals

**Lecture 8**: Markov processes. Gibbs sampling. Data augmentation. Probit regression. Finite mixture of normals.

**Read**: BDA Ch. 11 | Slides

**Code**: Simulating Markov Chains

**Lecture 9**: Metropolis-Hastings. Metropolis-Hastings within Gibbs. Heteroscedastic regression.

**Read**: BDA Ch. 11 | Slides

**Lab 3**: Gibbs sampling for the normal model, mixture of normals and probit regression.

Lab 3 | Rainfall data.

#### Module 4 - Model Inference, Variable Selection and Flexible Models

**Lecture 10**: Bayesian model inference, selection and averaging.

**Read**: BDA Ch. 7 | Slides

**Code**: Comparing models for count data

**Lecture 11**: Bayesian variable selection. Model checking.

**Read**: BDA 6.1-6.4 | Slides

**Lecture 12**: Hierarchical models. Regularized regression with Gibbs sampling. RStan demo.

**Read**: BDA Ch 5.1-5.5 and 15.1-15.5 | Slides

**Code**: Gaussian processes | RStan - Bernoulli model | RStan - Logistic regression with random effects |

RStan - Poisson model

**Lab 4**: Metropolis-Hastings for Poisson regression.

Lab 4 | eBay data.

#### Literature

**Bayesian Data Analysis**by Gelman, Carlin, Stern, och Rubin, Chapman & Hall, Third edition. The book's web site can be found here.- My
**slides**.

#### Examination

The examination for the course Bayesian Learning, 6hp, consists of

- written reports on the four computer labs (2 hp)
- individual written report on a project that applies Bayesian methods for data analysis (4hp)

**Deadline**for project proposal:

**TBA**.

**Deadline**for final project report:

**TBA**.

Here are some guidelines for the proposal and the project (nevermind the dates in that document).

Here are some very good example projects from previous years: Example1 | Example2

Submit proposal and final report in LISAM.

#### Bugs code

- BernBeta.R Bernoulli model
- BernBetaHierarchy.R Bernoulli model with estimated prior hyperparameters
- HeightWeight.R Linear regression
- LogisticRegRandEffects.R Logistic regression with random effects

#### Other material

- Informative clickable chart with relations between distributions: http://www.johndcook.com/distribution_chart.html.
- Learning about the prior-to-posterior mapping in:
- Bernoulli model with Beta prior

Google Docs | RStudio manipulate - Normal model with normal prior.

Google Docs - Poisson model with Gamma prior

Google Docs - Normal model with Laplace prior.

Google Docs

- Bernoulli model with Beta prior
- A collection with hundreds of machine learning datasets.
- An old exam with solutions from a course Mattias Villani gave in 2007. Have a look at Question 2 early in the course.

Page responsible: Mattias Villani

Last updated: 2016-04-20