Neural Networks and Deep Learning

PhD Course, 6 credits, Autumn 2018.
Instructors: Michael Felsberg (ISY), Marco Kuhlmann (IDA)

The course is full. To be added to the waiting list for the course, send an email to Marco Kuhlmann stating your LiU-ID.

Over the past few years, neural networks have enjoyed a major resurgence in machine learning, and today yield state-of-the-art results in various fields. This course provides an introduction to deep neural network models, and surveys some the applications of these models in areas where they have been particularly successful. The course covers feedforward networks, convolutional networks, recurrent and recursive networks, as well general topics such as input encoding and training techniques. The course also provides acquaintance with some of the software libraries available for building and training deep neural networks.

Syllabus

Prerequisites

Gaps in these prerequisites may be filled by teacher-assisted self-study before the start of the course; contact the instructors for details.

Intended learning outcomes

On successful completion of the course you should be able to:

Contents

Basics of machine learning (regression, classification, numerical optimisation). Feedforward networks. Loss functions. Back-propagation training. Regularisation. Convolutional networks. Recurrent and recursive networks. Processing sequences, images, and hierarchical structures. Applications of neural networks. Current areas of research.

Course literature

The main book for the course is:

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.

Additional reading consists of excerpts from the following books:

Organisation

Lectures

The lectures present basic concepts and methods in neural networks and survey applications. Each lecture connects to one or several chapters in the book by Goodfellow, Bengio, and Courville (GBC). The course schedule indicates which chapters you should have read before each lecture.

Lab sessions

The lab sessions give you practical experience in implementing, training, and evaluating neural networks using existing software libraries. There are a total of four labs:

Lab 0 is a preparatory lab that you do on-site. For each of the remaining labs there is an on-site introduction by an instructor, but most of the actual work is self-scheduled.

In order to do labs 1–3, you will need to bring your own computer and install the following software: Python 3, Jupyter Notebook, NumPy, TensorFlow, Keras. It is a good idea to install this software before the start of the course, but you can also get help with the installation during lab 0.

Research seminars

In the seminar sessions each student presents an article reporting current research on neural networks and their applications. You are encouraged to choose an article that is relevant for your own research. Instructions for presenters and schedule will be posted here after the start of the course.

The following links may be helpful when deciding upon a research article to present:

Individual project

The final part of the course is an individual project that you choose based on your own research interests. We anticipate the typical project to be one where you apply neural networks to solve a concrete problem related to your thesis work. Here is how it works:

Examination

The examination for the course consists of three parts:

You can choose to be examined on any combination of these parts. To be examined on the project part, you will have to submit a project plan (deadline, see below). If you do not submit a project plan, we will assume that you only want to be examined on the lab assignments and the seminars.

Deadlines for the examination:

Schedule

The lectures, lab sessions, and seminars take place in the B-building and the E-building at Campus Valla.

DayBefore sessionSession
2/10 10:15–12 Read GBC, chapters 1–5 Lecture 1: Basics of Machine Learning.
Location: Ada Lovelace
2/10 13:15–17 Lab session 0.
Location: John von Neumann
9/10 10:15–12 Read GBC, chapters 6–8 Lecture 2: Feedforward Networks.
Location: Ada Lovelace
9/10 13:15–17 Read instructions for lab 1 Lab session 1.
Location: John von Neumann
16/10 10:15–12 Read GBC, chapter 10 Lecture 3: Recurrent and Recursive Networks.
Location: Ada Lovelace
16/10 13:15–17 Read instructions for lab 2 Lab session 2.
Location: John von Neumann
23/10 10:15–12 Skim Goldberg Lecture 4: Applications in NLP.
Location: Ada Lovelace
23/10 13:15–17 Backup lab session.
Location: John von Neumann
30/10 10:15–12 Read GBC, chapter 9 Lecture 5: Convolutional Networks.
Location: Alan Turing
6/11 10:15–12 Lecture 6: Applications in Computer Vision.
Location: Alan Turing
6/11 13:15–17 Read instructions for lab 3 Lab session 3.
Location: Southfork
13/11 10:15–12 Lecture 7: Adversarial Learning.
Location: Alan Turing