$\newcommand{\E}{\mathbb{E}}$ $\newcommand{\R}{\mathbb{R}}$

Deep Learning in Practice

Chapter 3: Interpretability

NB: turn on javascript to get beautiful mathematical formulas thanks to MathJax
NB2: an old raw text file is also available if you wish for a more compact summary



I - Visualization / Analysis (of a neural network trained)
II - Interpretability: societal impact and approaches
III - Issues related to datasets

I - Visualization / Analysis (of a neural network trained)


Analysis, invariances

Adversarial attacks

→ Edouard's slides
→ Pierre Stock's presentation slides

NB: in the following, most is not specific to deep learning, but applicable to ML in general

II - Interpretability: societal impact and approaches

Why interpretability is important: what is at stake

Example: medical diagnosis

Societal impact: "Weapons of Maths Destruction" by Cathy O'Neil

$\implies$ crucial: feedback (from people involved), explanability, right to contest/appeal
$\implies$ think twice about the impact of your algorithms before deploying them

Be responsible and careful

"With great power comes great responsability" (guess the source;)

Interpretability by design: "Explainable AI"

By breaking the pipeline into interpretable steps
Example: image captioning

Interpretability of data: causality

Growing field of machine learning

III - Issues related to datasets

Dataset poisoning

Possible to forge a dataset:




NB: unfairness might be more subtle than expected
eg: word2vec trained on Google News:

Definition 1: fairness by (un)awareness

Simplistic version: unawareness
[Fairness through awareness; Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Rich Zemel; ITCS 2012]

Definition 2: Equal opportunity / $\epsi$-fairness

[Equality of Opportunity in Supervised Learning; Moritz Hardt, Eric Price, Nathan Srebro; NIPS 2016]
$\epsi$-fairness: same but approximately:

Definition 3: same distribution (of outputs / of errors) : group-based

Principle: probability of outcome (or success) should not depend (or not much) on the sensitive attribute

Example: study of main commercial face classification softwares, tested on a grid of different ages/genders/etc bins (check the performance on each subset: young white males, adult asian women, etc.) [Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification; Joy Buolamwini, Timnit Gebru; 1st Conference on FAT, 2018 ]
$\newcommand{\hy}{\widehat{y}}$ 3 possible requirements (with the same notations as above: sensitive attribute $A$ to be independent of, prediction $\hY$, true label or value $Y$ to predict):

independence $\hY$ independent of $A$ $\forall a, a', \hy, \;\;\;\;\;\;\;\; p(\hY=\hy|A=a) \;=\; p(\hY=\hy|A=a')$ outcome proba indep(group/sensitive info)
separation $\hY$ independent of $A$ when $|Y$ $\forall a, a',y,\hy, \;\;\;\;\;p(\hY=\hy|A=a,Y=y) \;=\; p(\hY=\hy|A=a',Y=y)$ $A$ doesn't influence distribution knowing skills : Equalized odds
sufficiency $Y$ independent of $A$ when $|\hY$ $\forall a, a',y,\hy, \;\;\;\;\; p(Y=y|A=a,\hY=\hy) \;=\; p(Y=y|A=a',\hY=\hy)$ $A$ doesn't influence the error distribution $y|\hy$

→ variations: do not require strict equality, but |difference| $< \epsi$, or ratio of probabilities $< 1+ \epsi$

NB: these group-based definitions are incompatible (if A and Y are correlated, you can't have any 2 of these independences at once)

Definition 4: Causality (Counterfactual fairness)

[Counterfactual Fairness; Matt J. Kusner, Joshua R. Loftus, Chris Russell, Ricardo Silva; NIPS 2017]


Type 1 [before training]: pre-process data, to remove sensitive data
Type 2 [while training]: enforce fairness while optimizing
Type 3 [after training]: at post-processing: change thresholds/biases
Example of type 2 with adversarial approach:
or enforce (soft, relaxed) constraints explicitely.

Example of type 1 :

Differential privacy

[NB: in French: "privacy" = "confidentialité"]

Issues regarding privacy

Why care about privacy? Isn't anonymization sufficient?
Netflix prize, 2007:
Why care if no dataset sharing?
If you (e.g., Google) train an algorithm on your client database (containing private data) and provide the trained algorithm to all clients as a service: it might be possible to extract private data (of other clients) from it

Queries on a database:

$\epsi$-differentiable privacy

Formalization of the amount of noise needed to be added to query answers to keep privacy, i.e. not be able to distinguish a dataset from the same dataset + one more element : $\epsi$-differential privacy
To go further:

Example of privacy-preserving pipeline

Example of advanced ML pipeline taking into account privacy:
[Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data; Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, Kunal Talwar; ICLR 2017]

Back to the main page of the course

Valid HTML 4.0 Transitional