TP4-M1ProML-Correction
Posted on Tue 30 April 2019 in posts
TP : regression et autoencoder¶
Le but de ce TP est de voir les deux modèles suivants
- la régression par réseau de neurone
- l'autoencoder
Régression¶
On a vu comment classifier des données par l'utilisation des réseaux de neurones. Ici on va voir rapidement comment il est possible de régresser des fonctions.
Soit la fonction suivante
In [21]:
import numpy as np
import matplotlib.pyplot as plt
X = np.random.random(5000)*20-10
y = (0.5+5*X+0.3*X**2-0.1*X**3)
plt.plot(X,y,'.')
plt.show()
In [22]:
from sklearn.neural_network import MLPRegressor
mlp1 = MLPRegressor(hidden_layer_sizes=(200,),batch_size=32,alpha=0,learning_rate_init=0.01,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 100
loss = np.zeros(TMAX)
for t in range(TMAX):
mlp1.fit(X.reshape(5000,1),y.reshape(5000,))
loss[t] = mlp1.loss_
In [23]:
plt.loglog(loss)
Out[23]:
In [24]:
xx = np.linspace(-10,10,100)
yp = mlp1.predict(xx.reshape(100,1))
yt = (0.5+5*xx+0.3*xx**2-0.1*xx**3)
In [25]:
plt.plot(xx,yp.reshape(100,1),'.')
plt.plot(xx,yt.reshape(100,1),'.')
plt.show()
In [26]:
# Let's put some noise
Ns = 10000
X = np.random.random(Ns)*20-10
y = (0.5+5*X+0.3*X**2-0.1*X**3) + 5*np.ones(X.shape)*np.random.normal(size=X.shape)
plt.plot(X,y,'.')
plt.show()
In [27]:
mlp1 = MLPRegressor(hidden_layer_sizes=(100,),batch_size=32,alpha=0,learning_rate_init=0.01,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 100
loss = np.zeros(TMAX)
for t in range(TMAX):
mlp1.fit(X.reshape(Ns,1),y.reshape(Ns,))
loss[t] = mlp1.loss_
In [32]:
plt.plot(loss)
Out[32]:
In [33]:
xx = np.linspace(-10,10,100)
yp = mlp1.predict(xx.reshape(100,1))
yt = (0.5+5*xx+0.3*xx**2-0.1*xx**3)
In [34]:
plt.plot(xx,yp.reshape(100,1),'.')
plt.plot(xx,yt.reshape(100,1),'.')
plt.show()
AutoEncoder¶
pour faire un autoencoder, il suffit d'utiliser soit la classe MLPClassifier (pour des variables binaires) soit la clase MLPRegressor pour des variables continues. On commencera par regarder le cas d'un autoencoder simple (binaire) sur MNIST. On verra ensuite comment on peut faire un "denoising autoencoder" afin de débruiter automatiquement une image contenant du bruit gaussien.
VOtre travail:
Première partie¶
- charger la bibliothèque MNIST
- Utiliser la classe MLPClassifier pour faire un autoencoder. Question : comment faut-il appeler la fonction 'fit' ?
- On plottera la loss de l'autoencoder en fonction du nombre d'itération.
- Afin de voir ce qui se passe, on affichera à chaque itération une image qui a été encodée puis décodée
- Finalement, vous ferez des tests en variant le nombre de variables cachées afin de trouver quel est le nombre optimal.
In [105]:
import pickle
import gzip
import numpy as np
import matplotlib.pyplot as plt
import numpy as np
f = gzip.open('../mnist.pkl.gz', 'rb')
u = pickle._Unpickler(f)
u.encoding = 'latin1'
p = u.load()
train_set, valid_set, test_set = p
In [106]:
from sklearn.neural_network import MLPRegressor
AE1 = MLPRegressor(solver='adam', hidden_layer_sizes=(100,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
# AE1 = MLPRegressor(solver='sgd',learning_rate_init=0.1,hidden_layer_sizes=(200,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 10
loss = np.zeros(TMAX)
for t in range(TMAX):
AE1.fit(train_set[0],train_set[0])
print(t,AE1.loss_)
loss[t] = AE1.loss_
In [107]:
f, ax = plt.subplots(2,2)
print("Resultat sur le train")
ax[0,0].imshow(train_set[0][0,:].reshape(28,28))
ax[0,1].imshow(AE1.predict(train_set[0][:1,:]).reshape(28,28))
print("Resultat sur le test")
ax[1,0].imshow(test_set[0][0,:].reshape(28,28))
ax[1,1].imshow(AE1.predict(test_set[0][:1,:]).reshape(28,28))
Out[107]:
In [108]:
# Discretisation
x_train_dis = (train_set[0] > 0.3)*1
x_test_dis = (test_set[0] > 0.3)*1
In [109]:
from sklearn.neural_network import MLPClassifier
AE1_Dis = MLPClassifier(solver='adam', hidden_layer_sizes=(100,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
# AE1 = MLPRegressor(solver='sgd',learning_rate_init=0.1,hidden_layer_sizes=(200,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 10
loss = np.zeros(TMAX)
for t in range(TMAX):
AE1_Dis.fit(x_train_dis,x_train_dis)
print(t,AE1_Dis.loss_)
loss[t] = AE1_Dis.loss_
In [110]:
f, ax = plt.subplots(2,2)
print("Resultat sur le train")
ax[0,0].imshow(x_train_dis[0,:].reshape(28,28))
ax[0,1].imshow(AE1_Dis.predict(x_train_dis[:1,:]).reshape(28,28))
print("Resultat sur le test")
ax[1,0].imshow(x_test_dis[0,:].reshape(28,28))
ax[1,1].imshow(AE1_Dis.predict(x_test_dis[:1,:]).reshape(28,28))
Out[110]:
In [120]:
# Test sur la première époque, comment évolue la reconstruction
from sklearn.neural_network import MLPClassifier
AE2_Dis = MLPClassifier(solver='adam', hidden_layer_sizes=(100,),batch_size=50,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
idx_plt_x = 0
idx_plt_y = 0
idx = np.random.choice(train_set[0].shape[0])
plt.imshow(x_train_dis[idx,:].reshape(28,28))
plt.show()
f, ax = plt.subplots(4,5,figsize=(10,5))
s_mb = 50
n_mb = int(50000/s_mb)
for mb in range(n_mb):
AE2_Dis.fit(x_train_dis[mb*s_mb:(mb+1)*s_mb,:],x_train_dis[mb*s_mb:(mb+1)*s_mb,:])
#print(AE2_Dis.loss_)
if(mb<200)&(mb%10==0):
pr = AE2_Dis.predict(x_train_dis[idx:(idx+1),:])
ax[idx_plt_y,idx_plt_x].imshow(pr.reshape(28,28))
idx_plt_x += 1
if(idx_plt_x==5):
idx_plt_y += 1
idx_plt_x = 0
plt.show()
In [121]:
plt.plot(loss)
Out[121]:
Test du nombre de variables cachées¶
(attention c'est long)
In [87]:
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import mean_squared_error, log_loss
hidd_conf = [10,20,50,100,200]
loss = np.zeros((len(hidd_conf),TMAX))
loss_t = np.zeros((len(hidd_conf),TMAX))
idx_xp = 0
for n_hidd in hidd_conf:
AE1_Dis = MLPClassifier(solver='adam', hidden_layer_sizes=(n_hidd,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
# AE1 = MLPRegressor(solver='sgd',learning_rate_init=0.1,hidden_layer_sizes=(200,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 10
for t in range(TMAX):
AE1_Dis.fit(x_train_dis,x_train_dis)
print(t,AE1_Dis.loss_)
loss[idx_xp,t] = mean_squared_error(x_train_dis,AE1_Dis.predict(x_train_dis))
loss_t[idx_xp,t] = mean_squared_error(x_test_dis,AE1_Dis.predict(x_test_dis))
idx_xp += 1
In [92]:
f, ax = plt.subplots(1,5,figsize=(15,5))
for i in range(5):
ax[i].plot(loss[i,:])
ax[i].plot(loss_t[i,:])
plt.show()
for i in range(5):
plt.plot(loss_t[i,:])
Seconde Partie¶
Ici, on va chercher à construire un autoencoder permettant de débruiter les images du jeu de données appris. On gardera donc MNIST, mais on appliquera la transformation suivante permettant de rajouter un bruit gaussien.
In [97]:
noise_factor = 0.3
x_train_noisy = x_train_dis + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=train_set[0].shape)
x_test_noisy = x_test_dis + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=test_set[0].shape)
x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)
In [98]:
plt.imshow(x_train_noisy[0,:].reshape(28,28))
Out[98]:
En utilisant le jeu de données bruités, essayer de trouver une façon d'utiliser l'autoencoder pour automatique débruiter les images. Vous vérifierez votre procédure
In [102]:
from sklearn.neural_network import MLPRegressor
AE1 = MLPClassifier(solver='adam', hidden_layer_sizes=(200,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
# AE1 = MLPRegressor(solver='sgd',learning_rate_init=0.1,hidden_layer_sizes=(200,),batch_size=32,alpha=0,max_iter=1,warm_start=True, activation='relu',n_iter_no_change=200)
TMAX = 15
loss = np.zeros(TMAX)
loss_t = np.zeros(TMAX)
for t in range(TMAX):
AE1.fit(x_train_noisy,x_train_dis)
print(t,AE1.loss_)
loss[t] = mean_squared_error(x_train_dis,AE1.predict(x_train_noisy))
loss_t[t] = mean_squared_error(x_test_dis,AE1.predict(x_test_noisy))
In [103]:
f, ax = plt.subplots(2,2)
print("Resultat sur le train")
ax[0,0].imshow(x_train_noisy[0,:].reshape(28,28))
ax[0,1].imshow(AE1.predict(x_train_noisy[:1,:]).reshape(28,28))
print("Resultat sur le test")
ax[1,0].imshow(x_test_noisy[0,:].reshape(28,28))
ax[1,1].imshow(AE1.predict(x_test_noisy[:1,:]).reshape(28,28))
Out[103]:
In [104]:
plt.plot(loss)
plt.plot(loss_t)
Out[104]:
In [ ]: