Probabilistic Generative Models: Schedule

Lecture 1

Outline:

• Course outline: typology of models (locally normalized models, globally normalized models, implicit models), methods (stochastic approximation, variational methods, ….)

• Probability remdinders

• Monte-Carlo estimation and importance sampling

• Sigmoid belief networks (SBN)

• Gaussian mixture models (GMM)

Code examples:

• The important sampling example we saw in the course is from here. You can dowload my Python+Pytorch implementation here: as a notebook or as a html file.

• Example of a GMM trained via gradient descent: as a notebook or as a html file.

Lecture 2

• Basics of convex analysis (notes)  —  (correction for a few exercises)

• Jensen's inequality

• Fenchel (bi)conjugates

• Expectation-Maximization (EM) and Mean Field Theory (slides)

• EM application: GMM (notes)

• Variational methods application: upper and lower bounds for SBNs (notes)

For more details about convex analysis and optimization, the the Convex Optimization book by Stephen Boyd and Lieven Vandenberghk is available online.

Lecture 3

• Globally normalized models and Boltzmann machines

• Variational methods application: upper and lower bounds for restricted Boltzmann machines

• MCMC Sampling: Metropolis-Hastings and Gibbs sampling

• MCMC application: Sigmoid belief networks and Boltzmann Machines

• MCMC exercise: download notebook or view as html

Lecture 4

• Variational Auto-Encoders

• Reparameterization trick

• Score function estimator

• Differentiable approximate sampling

Lecture 5

• Change of variable theorem

• Normalizing flows

• End of lecture on the logdet term for real NVP flows: page 1 page 2 page 3

• Normalzing flows examples:

  • Linear flows: download notebook or view as html

  • NICE flows: download notebook or view as html