Dropout Neural Networks in Machine Learning with Python
Hello learners, today we are going to learn a very interesting topic that is: Dropout Neural Networks. We all know that Neural Networks are notorious for over-fitting the data. Neural Networks, if not provided with enough input data are bound to overfit. We certainly want to avoid this situation. The techniques to avoid these errors are Regularization techniques and Dropout regularization is one of the techniques.
Here we are using Keras API from the TensorFlow deep learning library of Python for Dropout Neural Networks.
When a Neural Network is getting trained, each layer has some dependence on an input. It often happens that some layers become overly depended on few of the inputs. Dropout is a technique where neurons are selected randomly to be ignored during the cycle. A certain specified percentage of neurons will be dropped in each update cycle(the neurons to be dropped keep changing every cycle). This will reduce the over-dependence and hence prevent over-fitting. However, you should note that dropping out in each neural network does not necessarily mean improvement in its accuracy. You will often notice that that dropping out will sometimes lead to decrease in the NN’s accuracy.
You have a choice to apply the dropout function on any layer of the neural network, this also include the visible(input layer). Each time with the dropout function you will need to specify the fraction of neurons you to drop. For example: If one of my hidden layers contains 10 neurons and my dropout fraction is set to 0.3 , then in each cycle 3 random neurons from the 10 will be ignored
Following we will see the implementation of dropout in our neural network.
model = Sequential() model.add(Dense(13,activation='relu',input_shape=(n_cols_2,))) model.add(Dense(20,activation='relu')) model.add(Dense(13,activation='relu')) model.add(Dropout(0.3)) model.add(Dense(13,activation='sigmoid')) model.add(Dense(3,activation='softmax')) model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy']) model.fit(x=X_train,y=y_train,epochs=100,batch_size=10)
Here we can see that I have applied the Dropout function to the 2nd hidden layer i.e 30 percent of the 13 neurons will be dropped. It is always ideal to keep your dropout percentage between 20 to 50 percent.
Following is the entire code (I have considered the wine data, the wine is to be classified into one of the three categories)
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Activation from tensorflow.keras.optimizers import Adam from sklearn.datasets import load_wine from sklearn.model_selection import train_test_split from keras.utils import to_categorical import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline from sklearn.preprocessing import MinMaxScaler from tensorflow.keras.layers import Dropout data1 = pd.DataFrame(data= np.c_[df['data'], df['target']],columns= df['feature_names'] + ['target']) # making a dataframe from the data X = data1.drop('target',axis=1) #all the inputs train_y_2 = to_categorical(data1['target']) #all the outputs in one hot encoded form X_train, X_test, y_train, y_test = train_test_split(X,train_y_2,test_size=0.3,random_state=101) scaler = MinMaxScaler() X_train= scaler.fit_transform(X_train) X_test = scaler.transform(X_test) n_cols_2=X_train.shape model = Sequential() model.add(Dense(13,activation='relu',input_shape=(n_cols_2,))) model.add(Dense(20,activation='relu')) model.add(Dense(13,activation='relu')) model.add(Dropout(0.3)) model.add(Dense(13,activation='sigmoid')) model.add(Dense(3,activation='softmax')) model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy']) model.fit(x=X_train,y=y_train,epochs=100,batch_size=10) predictions=model.predict(X_test) predictionsfinal = np.around(predictions) predictionsfinal #the final prediction