Input similarity from the neural network perspective
G. Charpiat, N. Girard, L. Felardos, and Y. Tarabalka. Input Similarity from the Neural Network Perspective. In NeurIPS 2019 - 33th Annual Conference on Neural Information Processing Systems, Vancouver, Canada, Dec. 2019.
Context
This work takes place within a long-lasting collaboration with the Titane team (INRIA Sophia-Antipolis), within the ANR EPITOME about deep learning for satellite image analysis.
We chose to present this work in the portfolio in order to show the beneficial interactions between applications and core machine learning.
The study stems from noticing the unexpected, impressive accuracy of deep neural networks trained to register satellite images to cadaster maps: the network reaches much better accuracy than the noisy registration examples it learns from.
How is it even possible to produce results way more precise than the level of noise in the annotations of the training dataset? Could one explain and quantify such an auto-denoising phenomenon?
Contribution
The analysis shows that the neural net f takes advantage of the implicit similarity among samples, where the similarity is expressed in terms of: « two images x and x' are similar to the extent that an action aimed to modifying f(x) induces similar modifications on f(x') ». This topology, which turns out to be the so-called neural tangent kernel, enables the quantification of the auto-denoising phenomenon. Furthermore it also enables the fast and cheap estimation of the intrinsic sample density; this can be exploited for sample selection and learning regularization.
Impact
These results were presented both in the remote sensing community (as they show that noisy ground truth is not necessarily a problem) and in the core machine learning community, as they clarify the duality between the samples and the gradient of the network in these samples. These results were re-used in the ML community for different purposes, including similar image retrieval and sample influence quantification. The image registration code was re-used and re-adapted by the French Tax Office to detect undeclared swimming pool detection (about 40 millions € taxes back). Within the AO team, this work paved the way towards frugal AI, a main direction for future AO research.
Deep Statistical Solvers
Balthazar Donon, Wenzhuo Liu, Antoine Marot, Zhengying Liu, Isabelle Guyon, Marc Schoenauer. Deep Statistical Solvers. NeurIPS 2020, 34th Annual Conference on Neural Information Processing Systems, Dec. 2020.
Context
This work is a collaboration with EDF, RTE and IRT-SystemX, instanciated through two PhD Cifres, one with RTE (B. Donon) and one with IRT-SystemX (W. Liu).
We selected this paper in the portfolio in order to illustrate the growing usage of graph-NN and physical-informed approaches in the AO team.
The original motivation of this paper is to design a simulator for AC power grids. More generally, the question is to design a fast simulator for a physical system described by its behavioral law.
Contribution
The proposed approach uses the discrepancy w.r.t. the behavioral law as training criterion; it learns without samples, thus sparing the expensive construction of training datasets.
A Graph Neural Network architecture is used to enforce the simulation of the physical system in a spatially-invariant way.
Last, it is shown that the approach, called Deep Statistical Solvers, satisfies some Universal Approximation property for linear behavioral laws; in practice, it reaches an arbitrary accuracy while being faster by several orders of magnitude than the reference Newton-Raphson method used in the field.
Impact
This work was amongst the first ones within the AO team to follow the PINNs paradigm (Physically-Informed Neural Networks) in order to learn directly from physics laws instead of samples.
This paved the way for further ways to incorporate physical knowledge inside machine learning tasks, and in particular for further developments in ML/ODE (PhD Matthieu Nastorg, 2024, in a collaboration with IFP-EN and Safran) to solve fluid mechanics problems on irregular meshes using graph-NN, reinforcing the Physics-ML cross-disciplinary axis of the lab, and also leading to the creation of a start-up company (by PhD students) with INRIA support. This line of work is appreciated in the AI4Science community and is leading to various international invitations.
Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
A. Decelle, C. Furtlehner, and B. Seoane. Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines. NeurIPS 2021, 35th Annual Conference on Neural Information Processing Systems, Dec. 2021.
Context
This is joint work with the University Computense, Madrid. We chose this article in order to illustrate how statistical physics can be used to study training dynamics in machine learning.
Restricted Boltzman Machines (RBMs), at the crossroad of generative ML and statistical physics, are neural networks with a hidden layer that encode the distribution of data in the form of a Boltzmann distribution. This distribution is defined by an energy function whose parameters are, in principle, interpretable, which can be useful for many scientific applications. In general, this generative model is sampled via Monte Carlo Markov chains and Gibbs sampling. This sampling procedure is also used during training. The topic of this article is to study the different learning regimes and in particular the influence of the sampling procedure on them.
Contribution
The article establishes that there are two distinct learning regimes. The key factor is the relaxation time of the Markov chain. Either it is less than the training time, and the trained RBM corresponds to an equilibrium model, where the trained parameters of the RBM can be meaningfully interpreted. Or it is greater than the training time; the RBM then encodes a dynamic process (a non-equilibrium model); it can still be used to generate data (by setting the sampling horizon to that used during training), but its parameters are no longer interpretable. This article provides experimental evidence for these assertions; the theoretical analysis (considering the general case of energy-based models) is provided in another article by the same authors, published at ICML 2023. The contribution is to clarify the uses and abuses of RBMs in a wide range of applications.
Impact
The paper has been presented at many workshops in the physics community, where the use of tools like RBM is widespread. It has attracted a lot of interest as it clarifies the use of RBM in physics terms and has given us additional motivations to develop genuine equilibrium RBM learning algorithms in various settings.
Toward Job Recommendation for All
Guillaume Bied, Solal Nathan, Elia Perennes, Morgane Hoffmann, Philippe Caillou, Bruno Crépon, Christophe Gaillac, Michèle Sebag. Toward Job Recommendation for All. IJCAI 2023.
Context
This is a collaboration with ENSAE and Pôle Emploi / France Travail, chosen to illustrate the AI-for-Good axis of the AO team.
This paper reports on a 4 year joint research (Vadore Dataia project), aimed to build a real-world recommendation system for job seekers, exploiting the wealth of proprietary data of Pôle Emploi. The difficulty lies in the complexity and confidentiality of the data, in the pluridisciplinary collaboration between machine learners and economists and their different priorities, in the algorithmic (response time for hundred thousands job seekers and dozen thousands job offers) and ethical (see below) requirements.
Contribution and societal impact
A first challenge was to publish the paper, due to the limits of comparative validation: the Pôle Emploi data cannot be made public for obvious reasons; the only public comparable data (RecSys Challenge 2017) differ in significant respects from our data (e.g. the geographic information is removed to enforce data privacy).
A second challenge concerns the acceptability of the algorithm. Campaigns in-the-field on 100,000 job seekers were conducted by Pôle Emploi in Feb. 2022 and June 2023, to validate the relevance and acceptability of the recommendations.
A complementary study concerns the bias of the recommendations, compared with those of the real data (hirings and applications), regarding the gender. The extensive comparison shows that the average gender effect (controlled from the individual preferences, and skills) is same for the recommendations as for the hirings and applications.
National report on AI
Report "Donner un sens à l'Intelligence Artificielle" ("Give Meaning to AI"), known as "Rapport Villani".
Context
In September 2017, Prime Minister Edouard Philippe commissioned Cédric Villani, mathematician (Field Medal 2010) and En Marche MP, to write a report recommending a national strategy for the development of Artificial Intelligence (AI). Cédric Villani was surrounded by a multi-disciplinary team, including several members of the staff of the French National Digital Council (CNNum) and an AI computer scientist, Marc Schoenauer, member of LISN (A&O / TAU team, joint with Inria).
Role
The other members of the Villani commission (the Secretary General of CNNum, a lawyer, an economist and a political scientist) included just one other computer scientist, Bertrand Rondepierre, from DGA (Directorate General of Armaments), a recent graduate from the MVA Master program.
Marc Schoenauer, who has been working in the field since the late 80s, and was President of the AFIA (Association Française pour l'Intelligence Artificielle, French AI Association) from 2002 to 2004, thus played the role of AI expert in this commission, both from a technical point of view and from the point of view of the French landscape in the field - a sort of right-hand man to Cédric Villani for the drafting of the report.
Impact
The Villani report was officially presented to President Macron on March 28, 2018 at the Collège de France, and many of the recommendations it advocated have in fact inspired French AI policy in the years since in the form of the National Program in AI (PNIA). This gave rise, for example, to the so-called "3IA" institutes (Interdisciplinary AI Institutes), the AI Chairs program, and the Choose France program to bring back to France AI talent "lost" abroad. The report also supported the creation of the Jean Zay computing center, dedicated to AI and open to the entire French AI ecosystem. The report also highlighted AI trust issues, and defined certain priority sectors of activity: education, healthcare, mobility, agriculture and defense. The PNIA entered its second, acceleration phase a year ago, with an increase in the power of Jean Zay and a call for "AI clusters" to broaden the spectrum of funded institutes.