Neural Networks and Deep Learning
PhD Course, 3 + 3 credits, Autumn 2020.
Instructors: Michael Felsberg (ISY), Marco Kuhlmann (IDA)
This course was last given in Autumn 2020. There are currently no plans to give it again.
Over the past few years, neural networks have enjoyed a major resurgence in machine learning, and today yield state-of-the-art results in various fields. This course provides an introduction to deep neural network models, and surveys some the applications of these models in areas where they have been particularly successful. The course covers feedforward networks, convolutional networks, recurrent and recursive networks, as well as general topics such as input encoding and training techniques. The course also provides acquaintance with some of the software libraries available for building and training deep neural networks.
Syllabus
Prerequisites
- basic calculus (derivatives)
- basic linear algebra (matrices, vectors)
- basic probability and statistics
- programming experience (the lab series uses Python)
Gaps in these prerequisites may be filled by teacher-assisted self-study before the start of the course; contact the instructors for details.
Intended learning outcomes
On successful completion of the course you should be able to:
- explain different network architectures and how these are used in current applications
- implement, train, and evaluate neural networks using existing software libraries
- present and critically assess current research on neural networks and their applications
- relate the concepts and techniques introduced in the course to your own research
- plan and carry out a research project on neural networks within given time limits
Contents
Basics of machine learning (regression, classification, numerical optimisation). Feedforward networks. Loss functions. Back-propagation training. Regularisation. Convolutional networks. Recurrent and recursive networks. Processing sequences and images. Applications of neural networks. Current areas of research.
Course literature
The main book for the course is:
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Additional reading consists of excerpts from the following books:
- Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1996.
- Yoav Goldberg. Neural Network Models in Natural Language Processing. Morgan & Claypool, 2017.
- Simon O. Haykin. Neural Networks and Learning Machines. Third edition. Prentice Hall, 2008.
Organisation
Lectures
The lectures present basic concepts and methods in neural networks and survey applications. Each lecture connects to one or several chapters in the book by Goodfellow, Bengio, and Courville (GBC). The course schedule indicates which chapters you should have read before each lecture.
Labs
The labs give you practical experience in implementing, training, and evaluating neural networks using existing software libraries. There are a total of three labs:
- Lab 1: Feedforward Networks for Handwritten Digit Recognition
- Lab 2: Sequence Labelling with Deep Recurrent Networks
- Lab 3: Image Classification with Deep Convolutional Networks
To do lab 1 and lab 2, you will be able to use your own computer. There are two options: If you want to use your own software stack, you will need to install the following libraries: Python 3, Jupyter Notebook, NumPy, and PyTorch. We recommend installing this software in a virtual environment using Virtualenv or Anaconda. Please install the software before the start of the course. Alternatively, if you do not want to or are unable to install the necessary software on your own computer, you can also do the labs on Google Colab. To use the service, you will have to create a Google ID (or use an existing ID).
Lab 3 needs to be done using dedicated computers on campus.
Individual project
The second (optional) part of the course is an individual project that you formulate together with your PhD supervisor. We anticipate the typical project to be one where you apply neural networks to a concrete problem related to your thesis work. To be acceptable for the course, the project must contain a substantial amount of machine learning; in case of doubt about this, your supervisor is welcome to consult with us (the instructors).
Here is the process for the individual project:
- You write a project plan and submit it to your supervisor for feedback and clearance.
- You work on your project, seeking guidance and feedback from your supervisor.
- You summarise your findings in a conference-style oral presentation.
- You present your results at one of the seminars in the final week of the course.
To present your results, you must register for a seminar slot by sending an email to Marco. The email should contain:
- a proposed title for your presentation
- a statement indicating that your supervisor has approved your project
- the name and email address of your supervisor, in case we have any questions
The deadline for the registration of seminar slots is 8 December.
Examination
The examination for the course consists of two parts:
- lab assignments (3 credits)
- optional individual project (3 credits)
Based on whether or not you choose to do the project, you will get either 6 credits or 3 credits for this course.
Dates relevant to the examination:
- Due date for lab 1: 2020-11-12
- Due date for lab 2: 2020-11-19
- Deadline to register for a seminar slot: 2020-12-08
- Due date for lab 3: 2020-12-10
- Deadline for all lab assignments (late submissions, re-submissions): 2021-01-16
- Extra deadline: 2020-03-05
Lab assignments submitted after the first due date will be graded after the deadline for late submissions in January.
To meet a due date or deadline, it suffices to submit the assignment before 08:00 the first working day after that date.
Schedule
The lectures, lab sessions, and seminars take place in the B-building and the E-building at Campus Valla remotely via Zoom.
The lectures from the 2019 session have been videotaped, and are available via this link (requires login with LiU-ID).
Day | Before session | Session |
---|---|---|
20/10 13:15–15 | Read GBC, chapters 1–5 | Lecture 0: Machine Learning Basics. Location: Zoom (passcode sent via e-mail) |
3/11 8:15–10 | Read GBC, chapters 6–8 | Lecture 1: Deep Feedforward Networks. Location: Zoom (same passcode as for lecture 0) |
5/11 13:15–17 | Read instructions for lab 1 | Lab session 1. Location: Scheduled video calls |
10/11 8:15–10 | Read GBC, chapter 10 | Lecture 2: Recurrent Neural Networks. Location: Zoom (same passcode as for lecture 0) |
12/11 13:15–17 | Read instructions for lab 2 | Lab session 2. Location: Scheduled video calls |
17/11 8:15–10 | Skim Goldberg | Lecture 3: Applications in NLP. Location: Zoom (same passcode as for lecture 0) |
26/11 8:15–10 | Watch the video; read GBC, chapter 9 |
Lecture 4: Convolutional Networks (slides, video). QA session in Michael’s Zoom room (URL sent via mail) |
1/12 8:15–10 | Watch the video | Lecture 5: Applications in Computer Vision (slides, video). QA session in Michael’s Zoom room (URL sent via mail) |
3/12 13:15–17 | Read instructions for lab 3 | Lab session 3 (group 1). Location: tba |
8/12 8:15–10 | Watch the video | Lecture 6: Adversarial Learning (slides, video). QA session in Michael’s Zoom room (URL sent via mail) |
11/1–15/1 | Seminars; details tba | |
14/1 13:15–16:15 | Seminar sessions. Location: Zoom |