Dropout Neural Networks in Machine Learning with Python

By Value ML

Hello learners, today we are going to learn a very interesting topic that is: Dropout Neural Networks. We all know that Neural Networks are notorious for over-fitting the data. Neural Networks, if not provided with enough input data are bound to overfit. We certainly want to avoid this situation. The techniques to avoid these errors are Regularization techniques and Dropout regularization is one of the techniques.

Here we are using Keras API from the TensorFlow deep learning library of Python for Dropout Neural Networks.

Dropout Regularization

When a Neural Network is getting trained, each layer has some dependence on an input. It often happens that some layers become overly depended on few of the inputs.  Dropout is a technique where neurons are selected randomly to be ignored during the cycle. A certain specified percentage of neurons will be dropped in each update cycle(the neurons to be dropped keep changing every cycle). This will reduce the over-dependence and hence prevent over-fitting. However, you should note that dropping out in each neural network does not necessarily mean improvement in its accuracy. You will often notice that that dropping out will sometimes lead to decrease in the NN’s accuracy.

You have a choice to apply the dropout function on any layer of the neural network, this also include the visible(input layer). Each time with the dropout function you will need to specify the fraction of neurons you to drop. For example: If one of my hidden layers contains 10 neurons and my dropout fraction is set to 0.3 , then in each cycle 3 random neurons from the 10 will be ignored

Following we will see the implementation of dropout in our neural network.

model = Sequential()
model.add(Dense(13,activation='relu',input_shape=(n_cols_2,)))
model.add(Dense(20,activation='relu'))
model.add(Dense(13,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(13,activation='sigmoid'))
model.add(Dense(3,activation='softmax'))
model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(x=X_train,y=y_train,epochs=100,batch_size=10)

Here we can see that I have applied the Dropout function to the 2nd hidden layer i.e 30 percent of the 13 neurons will be dropped. It is always ideal to keep your dropout percentage between 20 to 50 percent.

Following is the entire code (I have considered the wine data, the wine is to be classified into one of the three categories)

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense, Activation

from tensorflow.keras.optimizers import Adam

from sklearn.datasets import load_wine

from sklearn.model_selection import train_test_split

from keras.utils import to_categorical

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

from sklearn.preprocessing import MinMaxScaler

from tensorflow.keras.layers import Dropout

data1 = pd.DataFrame(data= np.c_[df['data'], df['target']],columns= df['feature_names'] + ['target'])  # making a dataframe from the data 

X = data1.drop('target',axis=1) #all the inputs

train_y_2 = to_categorical(data1['target']) #all the outputs in one hot encoded form

X_train, X_test, y_train, y_test = train_test_split(X,train_y_2,test_size=0.3,random_state=101)

scaler = MinMaxScaler()

X_train= scaler.fit_transform(X_train)

X_test = scaler.transform(X_test)

n_cols_2=X_train.shape[1]

model = Sequential()

model.add(Dense(13,activation='relu',input_shape=(n_cols_2,)))

model.add(Dense(20,activation='relu'))

model.add(Dense(13,activation='relu'))

model.add(Dropout(0.3))

model.add(Dense(13,activation='sigmoid'))

model.add(Dense(3,activation='softmax'))

model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])

model.fit(x=X_train,y=y_train,epochs=100,batch_size=10)

predictions=model.predict(X_test)

predictionsfinal = np.around(predictions)

predictionsfinal #the final prediction