$\newcommand{\E}{\mathbb{E}}$ $\newcommand{\R}{\mathbb{R}}$

Deep Learning in Practice

Chapter 2: Interpretability

NB: turn on javascript to get beautiful mathematical formulas thanks to MathJax



I - Visualization / Analysis (of a neural network trained)
II - Interpretability: societal impact and approaches
III - Issues related to datasets

I - Visualization / Analysis (of a neural network trained)

At the neuron level

At the layer level

With CCA (Canonical correspondence analysis), check whether the features developed in such layer are correlated with another set of explainable features (e.g., handmade).

The case of CNN

At the functional level (of a network already trained)

About optimization visualization:

By sub-task design: “explainable AI”

Cf below.

NB: in the following, most is not specific to deep learning, but applicable to ML in general

II - Interpretability: societal impact and approaches

Why interpretability is important: what is at stake

Example: medical diagnosis

Societal impact: "Weapons of Maths Destruction" by Cathy O'Neil

$\implies$ crucial: feedback (from people involved), explanability, right to contest/appeal
$\implies$ think twice about the impact of your algorithms before deploying them

Be responsible and careful

"With great power comes great responsability" (guess the source;)

Interpretability by design: "Explainable AI"

By breaking the pipeline into interpretable steps
Example: image captioning

Interpretability of data: causality

Growing field of machine learning

III - Issues related to datasets

Dataset poisoning

Possible to forge a dataset:




NB: unfairness might be more subtle than expected
eg: word2vec trained on Google News:

Definition 1: fairness by (un)awareness

Simplistic version: unawareness
[Fairness through awareness; Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, Rich Zemel; ITCS 2012]

Definition 2: Equal opportunity / $\epsi$-fairness

[Equality of Opportunity in Supervised Learning; Moritz Hardt, Eric Price, Nathan Srebro; NIPS 2016]
$\epsi$-fairness: same but approximately:

Definition 3: same distribution (of outputs / of errors) : group-based

Principle: probability of outcome (or success) should not depend (or not much) on the sensitive attribute

Example: study of main commercial face classification softwares, tested on a grid of different ages/genders/etc bins (check the performance on each subset: young white males, adult asian women, etc.) [Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification; Joy Buolamwini, Timnit Gebru; 1st Conference on FAT, 2018 ]
$\newcommand{\hy}{\widehat{y}}$ 3 possible requirements (with the same notations as above: sensitive attribute $A$ to be independent of, prediction $\hY$, true label or value $Y$ to predict):

independence $\hY$ independent of $A$ $\forall a, a', \hy, \;\;\;\;\;\;\;\; p(\hY=\hy|A=a) \;=\; p(\hY=\hy|A=a')$ outcome proba indep(group/sensitive info)
separation $\hY$ independent of $A$ when $|Y$ $\forall a, a',y,\hy, \;\;\;\;\;p(\hY=\hy|A=a,Y=y) \;=\; p(\hY=\hy|A=a',Y=y)$ $A$ doesn't influence distribution knowing skills : Equalized odds
sufficiency $Y$ independent of $A$ when $|\hY$ $\forall a, a',y,\hy, \;\;\;\;\; p(Y=y|A=a,\hY=\hy) \;=\; p(Y=y|A=a',\hY=\hy)$ $A$ doesn't influence the error distribution $y|\hy$

→ variations: do not require strict equality, but |difference| $< \epsi$, or ratio of probabilities $< 1+ \epsi$

NB: these group-based definitions are incompatible (if A and Y are correlated, you can't have any 2 of these independences at once)

Definition 4: Causality (Counterfactual fairness)

[Counterfactual Fairness; Matt J. Kusner, Joshua R. Loftus, Chris Russell, Ricardo Silva; NIPS 2017]


Type 1 [before training]: pre-process data, to remove sensitive data
Type 2 [while training]: enforce fairness while optimizing
Type 3 [after training]: at post-processing: change thresholds/biases
Example of type 2 with adversarial approach:
or enforce (soft, relaxed) constraints explicitely.

Example of type 1 :

Differential privacy

[NB: in French: "privacy" = "confidentialité"]

Issues regarding privacy

Why care about privacy? Isn't anonymization sufficient?
Netflix prize, 2007:
Why care if no dataset sharing?
If you (e.g., Google) train an algorithm on your client database (containing private data) and provide the trained algorithm to all clients as a service: it might be possible to extract private data (of other clients) from it

Queries on a database:

$\epsi$-differentiable privacy

Formalization of the amount of noise needed to be added to query answers to keep privacy, i.e. not be able to distinguish a dataset from the same dataset + one more element : $\epsi$-differential privacy
To go further:

Example of privacy-preserving pipeline

Example of advanced ML pipeline taking into account privacy:
[Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data; Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, Kunal Talwar; ICLR 2017]

Federated learning

When training on sensitive data that should not be shared, for instance:

Back to the main page of the course

Valid HTML 4.0 Transitional