Caio Filippo Corro

Deep learning for natural language processing

This is the website for the Deep Learning for Natural Language Processing (NLP) course (OPT11) of Master 2 AI.

About

Recently, neural networks trained end-to-end have obtained impressive results in many problems related to natural languages (e.g. machine translation). These deep learning techniques do not rely on manual feature extraction or rule-based systems. However, behind the scenes a large part of this success is due to the development of neural architectures that are able to handle structured inputs and outputs.

In this course, we will study how build neural networks for problems related to natural languages. Specifically, we will learn how to:

Moreover, we will develop a critical analysis of state-of-the-art NLP models:

Grading Scheme

Project (100%)

Contact

You can contact me at caio.corro@u-psud.fr, either in French or English, with a subject starting with [OPT11]. Please, do not worry about typos or not being overly formal enough (just treat your instructors and colleagues with the same respect you would like to be treated).

Lectures (2021-2022)

Lecture 1: Introduction to deep learning for NLP

[Download slides]
[Download notes on neural language models]

References:

Lecture 2: Reccurent neural networks

[Download slides]

References:

To go further:

Blog posts:

Lecture 3: Attention mechanism

[Download slides]

References:

The transformer architecture is based on “tricks” that where proposed in these papers:

To go further:

Blog post:

Project

The goal of the project is to build a neural language model based on a LSTM and explore different variants. There is a list of subjects you can explore below. Obviously, you must not explore everything: select a few topics so that you can report comparative results. The “project roadmap” deadline is there so I can validate if your project is ok or not. I strongly recommend to discuss with me on what you want to do (before deadlines), in class or by e-mail (however, note that I will probably not be able to discuss by email if the first contact is one or two days before a deadline).

The grading will focus on:

Deadlines:

Here is a list of topics/subjects:

References