$\newcommand{\E}{\mathbb{E}}$ $\newcommand{\R}{\mathbb{R}}$

Deep Learning in Practice

Chapter 7: Small data, weak supervision, robustness / Modeling: incorporating priors or physics

NB: turn on javascript to get beautiful mathematical formulas thanks to MathJax
NB2: this page is still being updated



I - Small data, weak supervision, robustness II - Modeling: incorporating priors or physics

I - Small data, weak supervision, robustness

Small data

Weak supervision, self-supervision

Multi-task and transfer learning

Data augmentation, synthetic data, invariances

→ Edouard's slides
related exercise

II - Modeling: incorporating priors or physics

This section is about modeling the task: exploiting known invariances, priors or physical properties

Curriculum learning (from easy task to harder task)

[Curriculum learning; Y Bengio, J Louradour, R Collobert, J Weston; ICML 2009]

Example of invariance enforcement by design: permutation invariance

Deep Sets

[Deep Sets; Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, Alexander Smola; NIPS 2017]

Examples of applications: when dealing with unordered set of points

Choosing physically meaningful metrics

Optimal transport (Sinkhorn approximation)

Metric learning: Siamese networks

[Signature Verification using a "Siamese" Time Delay Neural Network; Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sickinger and Roopak Shah; NIPS 1994]

Noisy data (can be seen as noise modeling)

Denoising auto-encoder

[Extracting and composing robust features with denoising autoencoders; Vincent, H. Larochelle Y. Bengio and P.A. Manzagol; ICML 2008]

What if noise is already present in the dataset, but unknown? (no denoised target available)

Incorporating physical knowledge: Example of learning dynamics equations

Example: weather forecast, with training data
Predict weather from various sensor fields (temperature, pressure, ...) on a grid

Range of possibilities: from using a pre-existing model to learning everything from scratch

Data assimilation

Equation known, just a few coefficients to estimate, or conditions to re-estimate regularly (e.g., if chaotic behavior)
$\implies$ statistical estimates, Kalman filter

Learning a Partial Differential Equation (real equation not known)

Incorporating invariances/symmetries of the problem

Knowing an equation that the solution has to satisfy : solving PDEs!

[DGM: A deep learning algorithm for solving partial differential equations; Justin Sirignano and Konstantinos Spiliopoulos; Journal of Computational Physics 2018]

Important example in chemistry

[Machine Learning of coarse-grained Molecular Dynamics Force Fields; Jiang Wang, Christoph Wehmeyer, Frank Noe', Cecilia Clementi]

Form of the solution known

[Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge; Emmanuel de Bezenac, Arthur Pajot, Patrick Gallinari; 2017]

In practice

Learn from real samples (usually few) or from simulations (goal: improve simulation speed, or learn where to focus for good accuracy...)

Deep for physic dynamics : learning and controlling the dynamics

Consider a dynamical system:
Koopman - von Neumann classical mechanics using a quantum mechanics formalism:
[Deep Dynamical Modeling and Control of Unsteady Fluid Flows; Jeremy Morton, Freddie D. Witherden, Antony Jameson, Mykel J. Kochenderfer; NIPS 2018]
Cf also Steven Brunton's lab page, e.g.:
Active topic:
cf workshop : Machine Learning for Computational Fuild and Solid Dynamics

Back to the main page of the course

Valid HTML 4.0 Transitional