Neural Networks and Deep Learning
PhD Course, 3 + 3 credits, Autumn 2019.
Instructors: Michael Felsberg (ISY), Marco Kuhlmann (IDA)
Over the past few years, neural networks have enjoyed a major resurgence in machine learning, and today yield stateoftheart results in various fields. This course provides an introduction to deep neural network models, and surveys some the applications of these models in areas where they have been particularly successful. The course covers feedforward networks, convolutional networks, recurrent and recursive networks, as well general topics such as input encoding and training techniques. The course also provides acquaintance with some of the software libraries available for building and training deep neural networks.
Syllabus
Prerequisites
 basic calculus (derivatives)
 basic linear algebra (matrices, vectors)
 basic probability and statistics
 programming experience in Python or MATLAB
Gaps in these prerequisites may be filled by teacherassisted selfstudy before the start of the course; contact the instructors for details.
Intended learning outcomes
On successful completion of the course you should be able to:
 explain different network architectures and how these are used in current applications
 implement, train, and evaluate neural networks using existing software libraries
 present and critically assess current research on neural networks and their applications
 relate the concepts and techniques introduced in the course to your own research
 plan and carry out a research project on neural networks within given time limits
Contents
Basics of machine learning (regression, classification, numerical optimisation). Feedforward networks. Loss functions. Backpropagation training. Regularisation. Convolutional networks. Recurrent and recursive networks. Processing sequences, images, and hierarchical structures. Applications of neural networks. Current areas of research.
Course literature
The main book for the course is:
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Additional reading consists of excerpts from the following books:
 Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1996.
 Yoav Goldberg. Neural Network Models in Natural Language Processing. Morgan & Claypool, 2017.
 Simon O. Haykin. Neural Networks and Learning Machines. Third edition. Prentice Hall, 2008.
Organisation
Lectures
The lectures present basic concepts and methods in neural networks and survey applications. Each lecture connects to one or several chapters in the book by Goodfellow, Bengio, and Courville (GBC). The course schedule indicates which chapters you should have read before each lecture.
Labs
The labs give you practical experience in implementing, training, and evaluating neural networks using existing software libraries. There are a total of four labs:
 Lab 0: Software Installation
 Lab 1: Feedforward Networks for Handwritten Digit Recognition
 Lab 2: Sequence Labelling with Deep Recurrent Networks
 Lab 3: Image Classification with Deep Convolutional Networks
Lab 0 is a preparatory lab that you do onsite. For each of the remaining labs there is an onsite introduction by an instructor, but most of the actual work is selfscheduled.
In order to do labs 0–2, you will need to bring your own computer and install the following software: Python 3, Jupyter Notebook, NumPy, TensorFlow, Keras. We recommend installing this software using Virtualenv or Anaconda. It is a good idea to install this software before the start of the course, but you can also get help with the installation during lab 0.
Individual project
The second (optional) part of the course is an individual project that you formulate together with your PhD supervisor. We anticipate the typical project to be one where you apply neural networks to a concrete problem related to your thesis work. To be acceptable for the course, the project must contain a substantial amount of machine learning; in case of doubt about this, your supervisor is welcome to consult with us (the instructors).
Here is the process for the individual project:
 You write a short (1 page) project plan and submit it to your supervisor for feedback and clearance.
 You work on your project, seeking guidance and feedback from your supervisor.
 You summarise your findings in a conferencestyle oral presentation.
 You present your results at one of the seminars in the final week of the course.
Examination
The examination for the course consists of two parts:
 lab assignments (3 credits)
 optional individual project (3 credits)
Based on whether or not you choose to do the project, you will get either 6 credits or 3 credits for this course.
Dates relevant to the examination:
 Due date for lab 1: 20191104
 Due date for lab 2: 20191118
 Due date for lab 3: 20191209
 Deadline for all lab assignments (late submissions, resubmissions): 20200118
Lab assignments submitted after the first due date will be graded after the deadline for late submissions in January.
To meet a due date or deadline, it suffices to submit the assignment before 08:00 the first working day after that date.
Schedule
The lectures, lab sessions, and seminars take place in the Bbuilding and the Ebuilding at Campus Valla.
Day  Before session  Session 

15/10 13:15–15  Read GBC, chapters 1–5  Lecture 0: Machine Learning Basics. Location: Ada Lovelace 
15/10 15:15–17  Install software  Lab session 0. Location: John von Neumann 
21/10 8:15–10  Read GBC, chapters 6–8  Lecture 1: Deep Feedforward Networks. Location: Ada Lovelace 
23/10 13:15–17  Read instructions for lab 1  Lab session 1. Location: John von Neumann 
5/11 8:15–10  Read GBC, chapter 10  Lecture 2: Recurrent and Recursive Networks. Location: Ada Lovelace 
7/11 13:15–17  Read instructions for lab 2  Lab session 2. Location: John von Neumann 
12/11 8:15–10  Skim Goldberg  Lecture 3: Applications in NLP. Location: Ada Lovelace 
14/11 13:15–17  Backup lab session. Location: John von Neumann 

21/11 13:15–15  Read GBC, chapter 9  Lecture 4: Convolutional Networks. Location: Ada Lovelace 
26/11 8:15–10  Lecture 5: Applications in Computer Vision. Location: Ada Lovelace 

28/11 13:15–17  Read instructions for lab 3  Lab session 3. Location: Olympen 
3/12 8:15–10  Lecture 6: Adversarial Learning. Location: Ada Lovelace 

14/1 8:15–10  Seminar session S1 (preliminary). Location: tbd 

15/1 10:15–12  Seminar session S2 (preliminary). Location: tbd 

16/1 13:15–17  Seminar session S3 (preliminary). Location: tbd 

17/1 10:15–12  Seminar session S4 (preliminary). Location: tbd 