Neural Networks and Deep Learning
PhD Course, 6 credits, Autumn 2018.
Instructors: Michael Felsberg (ISY), Marco Kuhlmann (IDA)
Over the past few years, neural networks have enjoyed a major resurgence in machine learning, and today yield stateoftheart results in various fields. This course provides an introduction to deep neural network models, and surveys some the applications of these models in areas where they have been particularly successful. The course covers feedforward networks, convolutional networks, recurrent and recursive networks, as well general topics such as input encoding and training techniques. The course also provides acquaintance with some of the software libraries available for building and training deep neural networks.
Syllabus
Prerequisites
 basic calculus (derivatives)
 basic linear algebra (matrices, vectors)
 basic probability and statistics
 programming experience in Python, Java or MATLAB
Gaps in these prerequisites may be filled by teacherassisted selfstudy before the start of the course; contact the instructors for details.
Intended learning outcomes
On successful completion of the course you should be able to:
 explain different network architectures and how these are used in current applications
 implement, train, and evaluate neural networks using existing software libraries
 present and critically assess current research on neural networks and their applications
 relate the concepts and techniques introduced in the course to your own research
 plan and carry out a research project on neural networks within given time limits
Contents
Basics of machine learning (regression, classification, numerical optimisation). Feedforward networks. Loss functions. Backpropagation training. Regularisation. Convolutional networks. Recurrent and recursive networks. Processing sequences, images, and hierarchical structures. Applications of neural networks. Current areas of research.
Course literature
The main book for the course is:
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Additional reading consists of excerpts from the following books:
 Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1996.
 Yoav Goldberg. Neural Network Models in Natural Language Processing. Morgan & Claypool, 2017.
 Simon O. Haykin. Neural Networks and Learning Machines. Third edition. Prentice Hall, 2008.
Organisation
Lectures
The lectures present basic concepts and methods in neural networks and survey applications. Each lecture connects to one or several chapters in the book by Goodfellow, Bengio, and Courville (GBC). The course schedule indicates which chapters you should have read before each lecture.
Lab sessions
The lab sessions give you practical experience in implementing, training, and evaluating neural networks using existing software libraries. There are a total of four labs:
 Lab 0: Software Installation
 Lab 1: Feedforward Networks for Handwritten Digit Recognition
 Lab 2: Sequence Labelling with Deep Recurrent Networks
 Lab 3: Image Classification with Deep Convolutional Networks
Lab 0 is a preparatory lab that you do onsite. For each of the remaining labs there is an onsite introduction by an instructor, but most of the actual work is selfscheduled.
In order to do labs 1–3, you will need to bring your own computer and install the following software: Python 3, Jupyter Notebook, NumPy, TensorFlow, Keras. It is a good idea to install this software before the start of the course, but you can also get help with the installation during lab 0.
Research seminars
In the seminar sessions each student presents an article reporting current research on neural networks and their applications. You are encouraged to choose an article that is relevant for your own research. Instructions for presenters and schedule will be posted here after the start of the course.
The following links may be helpful when deciding upon a research article to present:
 Five Years After the Deep Learning Revolution of Computer Vision
 A Primer on Neural Network Models for Natural Language Processing
Individual project
The final part of the course is an individual project that you choose based on your own research interests. We anticipate the typical project to be one where you apply neural networks to solve a concrete problem related to your thesis work. Here is how it works:
 You write a short (1 page) project plan and submit it to one of the instructors (deadline, see below).
 You get feedback on your project plan and make any required adjustments.
 You work on your project and summarise your findings in a report (think: short conference paper).
 You submit your report for grading (deadline, see below).
Examination
The examination for the course consists of three parts:
 lab assignments (1.5 credits)
 presentation and active participation in the research seminars (1.5 credits)
 individual project (3 credits)
You can choose to be examined on any combination of these parts. To be examined on the project part, you will have to submit a project plan (deadline, see below). If you do not submit a project plan, we will assume that you only want to be examined on the lab assignments and the seminars.
Deadlines for the examination:
 Lab assignments due: tbd (first half of November)
 Project plan due: tbd (first half of November)
 Project report due: 20190121 (8:00)
Schedule
The lectures, lab sessions, and seminars take place in the Bbuilding and the Ebuilding at Campus Valla.
Day  Before session  Session 

2/10 10:15–12  Read GBC, chapters 1–5  Lecture 1: Basics of Machine Learning. Location: Ada Lovelace 
2/10 13:15–17  Lab session 0. Location: John von Neumann 

9/10 10:15–12  Read GBC, chapters 6–8  Lecture 2: Feedforward Networks. Location: Ada Lovelace 
9/10 13:15–17  Read instructions for lab 1  Lab session 1. Location: John von Neumann 
16/10 10:15–12  Read GBC, chapter 10  Lecture 3: Recurrent and Recursive Networks. Location: Ada Lovelace 
16/10 13:15–17  Read instructions for lab 2  Lab session 2. Location: John von Neumann 
23/10 10:15–12  Skim Goldberg  Lecture 4: Applications in NLP. Location: Ada Lovelace 
23/10 13:15–17  Backup lab session. Location: John von Neumann 

30/10 10:15–12  Read GBC, chapter 9  Lecture 5: Convolutional Networks. Location: Alan Turing 
6/11 10:15–12  Lecture 6: Applications in Computer Vision. Location: Alan Turing 

6/11 13:15–17  Read instructions for lab 3  Lab session 3. Location: Southfork 
13/11 10:15–12  Lecture 7: Adversarial Learning. Location: Alan Turing 