Traffic Sign Classification with TensorFlow in Python

By Atharva Chourey

Hello everyone, welcome to Traffic Sign Classification tutorial using Tensorflow in Python. In this article, we will be focusing on how to classify traffic signs using deep learning(CNN). So here we are using a dataset with 43 different classes of images of signals. In self-driving cars, this is one of the major tasks to be performed.

Before we start, please install all the necessary libraries(mentioned below) and download the dataset from here: https://www.kaggle.com/tomerel/traffic-signs-pickled-dataset/data

Note: Please make sure your dataset and your script are in the same folder or otherwise specify the path in the code. Also, Jupyter notebook is preferred here.

Importing Libraries

In the below code all the necessary libraries are imported.

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import pandas as pd
import seaborn as sns
import pickle
import random

Loading Dataset

So here, Using pickle we will be loading the dataset into the train, valid and test parameters. The mode of loading the dataset is set to read binary(i.e. ‘rb’).

with open("./train.p", mode='rb') as training_data:
    train = pickle.load(training_data)
with open("./valid.p", mode='rb') as validation_data:
    valid = pickle.load(validation_data)
with open("./test.p", mode='rb') as testing_data:
    test = pickle.load(testing_data)

Next, we are dividing it into features and labels. Here you can also check the shape of train, test and validation data using shape.

X_train,y_train = train['features'],train['labels']
X_validation,y_validation = valid['features'],valid['labels']
X_test,y_test = test['features'],test['labels']

Image Visualization

Let’s take a random image from the training set to check its label.

i = np.random.randint(1, len(X_train))
plt.imshow(X_train[i])
y_train[i]

Output:

13 

 

Above we can see an image of a yield traffic sign with label 13.

So now, instead of taking a single image lets make a grid of 10*10 so that we can view 100 different images with their respective labels.

B_grid = 10
L_grid = 10

fig, axes = plt.subplots(L_grid, B_grid, figsize = (10,10))
axes = axes.ravel() 
n_training = len(X_train) 

for i in np.arange(0,B_grid*L_grid):
    index = np.random.randint(0,n_training)
   
    axes[i].imshow(X_train[index])
    axes[i].set_title(y_train[index], fontsize = 15)
    axes[i].axis('off')
plt.subplots_adjust(hspace=0.4)

Output:

Above we find various images with their labels.

Converting Images To Grayscale And Performing Normalization

Now, we will use sklearn to shuffle the data because we don’t want our network to learn the order of the images.

from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train)
X_validation, y_validation = shuffle(X_validation, y_validation)
X_test, y_test = shuffle(X_test, y_test)

And next, we are converting the images from RGB to grayscale by averaging them.

X_train_gray = np.sum(X_train/3,axis = 3,keepdims = True)
X_validation_gray = np.sum(X_validation/3,axis = 3,keepdims = True)
X_test_gray = np.sum(X_test/3,axis = 3,keepdims = True)

To check the shape of the data we can use shape keyword. This task is for you to perform.

So now let’s normalize the data by first subtracting it by 128 and then diving it by 128. Below code will normalize our dataset.

X_train_gray_norm = (X_train_gray - 128)/128
X_test_gray_norm = (X_test_gray - 128)/128
X_validation_gray_norm = (X_validation_gray - 128)/128

Here also you can check whether the whole data is converted or not using shape.

So finally, we will visualize the images that we have converted and normalized above. The first image is the grayscaled image, the second one is the coloured image and the final image is the normalized grayscale image which we will use to train our neural network.

i = random.randint(1, len(X_train_gray))
plt.imshow(X_train_gray[i].squeeze(), cmap = 'gray')
plt.figure()
plt.imshow(X_train[i])
plt.figure()
plt.imshow(X_train_gray_norm[i].squeeze(), cmap = 'gray')

Output:

Building CNN Model

So for building the CNN model, we have followed the following procedure:

Input-Image -> Convolution -> Pooling -> Flattening -> Neural Network -> Output

Using Keras we have imported CNN model and done convolution, pooling and flattening procedures.

In the below code we will add different layers to our CNN model.

For convolution layer, we have taken number of convolutions as 6 and size as (5, 5). We are using the relu activation function here. Next, we will add other dense layers as you can see below.

from tensorflow.keras import datasets, layers, models
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import AveragePooling2D
CNN = models.Sequential()

CNN.add(layers.Conv2D(6,(5,5),activation = 'relu', input_shape = (32,32,1)))
CNN.add(layers.AveragePooling2D())
CNN.add(layers.Flatten())
    
CNN.add(layers.Dense(120,activation = 'relu'))
CNN.add(layers.Dense(84,activation = 'relu'))
CNN.add(layers.Dense(43,activation = 'softmax'))
CNN.summary()

Output:

Using TensorFlow backend.

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 6)         156       
_________________________________________________________________
average_pooling2d (AveragePo (None, 14, 14, 6)         0         
_________________________________________________________________
flatten (Flatten)            (None, 1176)              0         
_________________________________________________________________
dense (Dense)                (None, 120)               141240    
_________________________________________________________________
dense_1 (Dense)              (None, 84)                10164     
_________________________________________________________________
dense_2 (Dense)              (None, 43)                3655      
=================================================================
Total params: 155,215
Trainable params: 155,215
Non-trainable params: 0
____________________________

CNN.compile(optimizer = 'Adam', loss='sparse_categorical_crossentropy',metrics = ['accuracy'])

Next, we are training the model with a batch size of 500 images(that we converted into grayscale and normalized), 10 epochs and verbose = 1 (which is the text we need to show during training).

history = CNN.fit(X_train_gray_norm,
                 y_train,
                 batch_size=500,
                 epochs =10,
                 verbose = 1,
                 validation_data = (X_validation_gray_norm,y_validation))

Output:

Train on 34799 samples, validate on 4410 samples 
Epoch 1/10 
34799/34799 [==============================] - 7s 210us/sample - loss: 2.8456 - accuracy: 0.2711 - val_loss: 2.3766 - val_accuracy: 0.3567
Epoch 2/10
34799/34799 [==============================] - 7s 189us/sample - loss: 1.4795 - accuracy: 0.5975 - val_loss: 1.4561 - val_accuracy: 0.5780
Epoch 3/10
34799/34799 [==============================] - 7s 188us/sample - loss: 0.8994 - accuracy: 0.7630 - val_loss: 1.0495 - val_accuracy: 0.6993
Epoch 4/10
34799/34799 [==============================] - 7s 189us/sample - loss: 0.6474 - accuracy: 0.8357 - val_loss: 0.9413 - val_accuracy: 0.7186
Epoch 5/10
34799/34799 [==============================] - 7s 188us/sample - loss: 0.5203 - accuracy: 0.8683 - val_loss: 0.8513 - val_accuracy: 0.7542
Epoch 6/10
34799/34799 [==============================] - 7s 195us/sample - loss: 0.4262 - accuracy: 0.8967 - val_loss: 0.7543 - val_accuracy: 0.7866
Epoch 7/10
34799/34799 [==============================] - 8s 236us/sample - loss: 0.3663 - accuracy: 0.9102 - val_loss: 0.7343 - val_accuracy: 0.7878
Epoch 8/10
34799/34799 [==============================] - 8s 233us/sample - loss: 0.3216 - accuracy: 0.9218 - val_loss: 0.7021 - val_accuracy: 0.8093
Epoch 9/10
34799/34799 [==============================] - 7s 209us/sample - loss: 0.2828 - accuracy: 0.9329 - val_loss: 0.7412 - val_accuracy: 0.8029
Epoch 10/10
34799/34799 [==============================] - 7s 205us/sample - loss: 0.2502 - accuracy: 0.9407 - val_loss: 0.7002 - val_accuracy: 0.8129

Assessing The Model Performance

We will now see the accuracy of our model on testing data.

score = CNN.evaluate(X_test_gray_norm, y_test)
print('Test Accuracy: {}'.format(score[1]))

Output:

Test Accuracy: 0.8225653171539307

history.history.keys()

Output:

dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

We will get the dictionary keys here so that we can use them for image visualization.

accuracy = history.history['accuracy']
val_accuracy = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

So now, we will see how the training and validation loss changes with the number of epochs.

epochs = range(len(accuracy))
plt.plot(epochs, loss, 'ro', label = 'Training loss')
plt.plot(epochs, val_loss, 'r', label = 'Validation loss')
plt.title('Training and validation loss')

Output:

Text(0.5, 1.0, 'Training and validation loss')

Now we will see Training and validation accuracy changes with the increase in epochs.

epochs = range(len(accuracy))
plt.plot(epochs, accuracy, 'ro', label = 'Training accuracy')
plt.plot(epochs, val_accuracy, 'r', label = 'Validation accuracy')
plt.title('Training and validation accuracy')

Output:

Text(0.5, 1.0, 'Training and validation accuracy')

Next, we will draw a heatmap using a confusion matrix. It will be a massive matrix since we have 43 different classes of images.

predicted_classes = CNN.predict_classes(X_test_gray_norm)
y_true = y_test

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_true, predicted_classes)
plt.figure(figsize = (25, 25))
sns.heatmap(cm, annot = True)

Output plot:

since it is a huge matrix the image is not much clear but you will get a similar matrix when you run the code.

Finally, we will see how our model predicts the labels of the images.

L = 5
B = 5

fig, axes = plt.subplots(L, B, figsize = (12, 12))
axes = axes.ravel()

for i in np.arange(0, L*B):
    axes[i].imshow(X_test[i])
    axes[i].set_title('Prediction = {}\n True = {}'.format(predicted_classes[i], y_true[i]))
    axes[i].axis('off')

plt.subplots_adjust(wspace = 1)

Output:

So here, we can see that most of the images are correctly predicted which means our model is successful. So we are able to do the traffic sign classification using TensorFlow in Python language.

Note: You can increase the accuracy by increasing the number of epochs and trying different layers in CNN. So try this while training the model.

Hope you liked this article.

Thank you for reading!!