This is the website for the Deep Learning for Natural Language Processing (NLP) course (OPT11) of Master 2 AI.
Recently, neural networks trained end-to-end have obtained impressive results in many problems related to natural languages (e.g. machine translation). These deep learning techniques do not rely on manual feature extraction or rule-based systems. However, behind the scenes a large part of this success is due to the development of neural architectures that are able to handle structured inputs and outputs.
In this course, we will study how build neural networks for problems related to natural languages. Specifically, we will learn how to:
Moreover, we will develop a critical analysis of state-of-the-art NLP models:
You can contact me at firstname.lastname@example.org, either in French or English, with a subject starting with [OPT11]. Please, do not worry about typos or not being overly formal enough (just treat your instructors and colleagues with the same respect you would like to be treated).
Lecture 1: Introduction to deep learning for NLP
Lecture 2: Reccurent neural networks
To go further:
Lecture 3: Attention mechanism
The transformer architecture is based on “tricks” that where proposed in these papers:
To go further:
The goal of the project is to build a neural language model based on a LSTM and explore different variants. There is a list of subjects you can explore below. Obviously, you must not explore everything: select a few topics so that you can report comparative results. The “project roadmap” deadline is there so I can validate if your project is ok or not. I strongly recommend to discuss with me on what you want to do (before deadlines), in class or by e-mail (however, note that I will probably not be able to discuss by email if the first contact is one or two days before a deadline).
The grading will focus on:
Here is a list of topics/subjects: