Deep Learning in Practice

MVA Master, January-March 2023

News : (reload for fresher news...)
- Contents remain accessible upon registration

The previous course (January-March 2022) can be found there; this course follows more or less the same lines, though not exactly.

General information about the course: presentation slides

To get access the recordings of this course, register here (NB: this is not the registration to the 2024 course, which is there.)

Requirements : having already followed a course about neural networks (this is an advanced deep learning course).
Typical mathematical notions used: differential calculus, Bayesian statistics, analysis, information theory.

Teaching team :

Most lectures: Guillaume Charpiat
Practical sessions: Matthieu Nastorg, Francesco Pezzicoli and Cyriaque Rousselot (incl. materials by Victor Berger, Alessandro Bucci, Loris Felardos and Wenzhuo Liu)

In a nutshell:

Chapter	Title	Summary	Notes	Videos	Exercises
1	Deep learning vs. classical ML and optimization	html	pdf	1 [2022]	Hyperparameter and training basics Environment and practical tips and tricks
2	Interpretability	html	pdf 1, 2	1, 2, 3, 4	Visualization with grad-CAM
3	Architectures	html	pdf 1, 2	1, 2, 3	Graph-NN
4	Small data: weak supervision, transfer, and incorporation of priors	html	pdf	1, 2	Transfer learning: jupyter notebook
5	Modeling: deep learning and physics (exploiting known invariances, priors or physical properties)	html	pdf 1, 2, + refs by A. Bucci	1, 2, 3	Learning dynamical systems (Credits: Alessandro Bucci)
6	Generative models (GAN, VAE, Normalizing Flows and Diffusion Models)	-	pdf	1, 2	Generative models
7	Modeling tasks and losses + Auto-ML / Auto-DeepLearning by Lisheng Sun-Hosoya and Romain Égelé from Isabelle Guyon's team	html	pdf, AutoML/MetaLearning slides, Neural Architecture Search slides	1, 2	-

Program :

January 24th : Deep learning vs. classical machine learning and optimization
→ Practical session: hyperparameters and training basics: instructions and the 2 problems of TP1: Pb 1, and Pb 2 with its dataset
→ Link to hand in TP1 (same password as usual)
→ Environment used in practical sessions
→ Practical tips and tricks to train neural networks
→ [2022] Course notes (pdf) (handwritten with drawings) and lesson summary (html)
→ [2022] Video recording: lecture [270MB], TP presentation [130MB]
January 31st : Interpretability (part 1: visualization and analysis)
→ Practical session: visualization with grad-CAM: notebook and test images
→ Link to hand in TP2 (same password as usual)
→ [2023] Course notes (pdf) (handwritten with drawings) and lesson summary (html)
→ [2023] Video recording: part 1 [500MB], part 2 [700MB]
February 7th : Interpretability (part 2: biases and issues with datasets)
→ [2023] Course notes (pdf) (handwritten with drawings) and lesson summary (html)
→ [2023] Video recording: part 3 [700MB], part 4 [400MB]
February 28th : Architectures
→ Practical session on graph-NN: notebook
→ Link to hand in TP3 (same password as usual)
→ [2023] Course notes (pdf) + supplementary notes from next session (handwritten with drawings) and lesson summary
→ [2023] Video recording: part 1 [500MB] (theory: prior, initialization, ...) and part 2 [500MB] (architecture zoo, attention, graph-NN and TP3 presentation)
March 7th : Small data: weak supervision, transfer, and incorporation of priors
→ Practical session on transfer learning: jupyter notebook
→ Link to hand in TP4
→ [2023] Course notes (pdf) (handwritten with drawings) and lesson summary
→ [2023] Video recording: part 1 [500MB] and part 2 [500MB]
March 14th : Modeling: deep learning and physics (exploiting known invariances, priors or physical properties)
→ Practical session on learning dynamical systems (notebook + image included by the notebook) (Credits: Alessandro Bucci), with its dedicated submission site
→ [2023] Course notes (pdf) + supplementary notes from next session (handwritten with drawings) and lesson summary
→ [2023] Video recording: part 1 [850MB] and part 2 [500MB]
March 21st : Generative models (GAN, VAE, Normalizing Flows and Diffusion Models)
→ Practical session on generative models, and its dedicated submission site
→ [2023] Course notes (pdf) on generative models (handwritten with drawings)
→ [2023] Video recording: part 1 [400MB] and part 2 + TP6 presentation [550MB]
March 28th : Modeling tasks and losses + Auto-ML / Auto-DeepLearning (guest talks by Lisheng Sun-Hosoya and Romain Égelé from Isabelle Guyon's team)
→ [2023] Video recordings: course on Modeling tasks and losses + presentation "AutoML via Bayesian Optimization and Meta-Learning" by Lisheng Sun-Hosoya, [400MB] and presentation "Neural Architecture Search" by Romain Égelé [260MB]
→ [2023] Course notes (pdf) about metrics (handwritten with drawings) + AutoML slides + Neural Architecture Search slides
→ Lesson summary about modeling tasks and losses and about learning from noisy data

Ressources: links to introduction to python, numpy, classical machine learning + online deep learning courses

Internship offers :

ours:
- Sampling 3D Molecular Distributions with Deep Neural Networks
- Learning dynamical systems with the right statistics: distribution of extreme events in complex physical systems
- Learning causality
- and lots of other very interesting topics, on demand (tiny neural network growth by spotting expressivity bottlenecks, etc.)
more from the TAU team (deep for population genetics, AutoML, etc.)

Back to the main page