Steel Surface Inspection in Keras | Python

By Jerrin Bright

Steel surface inspection is usually done by professionals and it is a very important inspection especially for social infrastructures. Large-scale inspection and position capturing by professionals in high acc are practically not possible. Moreover, it will be very expensive to rent such professionals. Thus, robots for such inspection systems increase the acc of the system and its consistency.

In this article, we are going to do the task of autonomous steel surface inspection using Xception Network is experimented with and analyzed in TensorFlow Keras API with complete Python code. A result of 98.61% in validation accuracy is obtained as a result of the built network. The workflow of this coming sections is as follows:

• Importing libraries
• Loading data
• Pre-processing the loaded data
• Building the network from the Xception Net
• Declaring the callback functions
• Compilation and Training
• Plotting for visualization

Happy Reading!!

LIBRARIES

Importing Keras TensorFlow and all the other required Python libraries.

import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import glob
import random
from matplotlib.patches import Rectangle
from lxml import etree
%matplotlib inline

LOAD DATA

In this section the tasks done are as follows:

• Storing all the images in a path file
• Extraction of all the XML files
• Extraction of all the Image files
• Checking the image to labels
• Storing the label names separately

image_path = glob.glob('../input/images/images/*/*.jpg')
xmls_path = glob.glob('../input/label/label/*.xml')
xmls_train = [p.split('/')[-1].split('.')[0] for p in xmls_path]
imgs_train = [img for img in image_path if (img.split('/')[-1].split)('.jpg')[0] in xmls_train]
xmls_path.sort(key=lambda x:x.split('/')[-1].split('.xml')[0])
imgs_train.sort(key=lambda x:x.split('/')[-1].split('.jpg')[0])
names = [x.split("/")[-2] for x in imgs_train]
names = pd.DataFrame(names,columns=['Types'])
from sklearn.preprocessing import LabelBinarizer

Class = names['Types'].unique()
Class_dict = dict(zip(Class, range(1,len(Class)+1)))
names['str'] = names['Types'].apply(lambda x: Class_dict[x])
lb = LabelBinarizer()
lb.fit(list(Class_dict.values()))
transformed_labels = lb.transform(names['str'])
y_bin_labels = []  

for i in range(transformed_labels.shape[1]):
    y_bin_labels.append('str' + str(i))
    names['str' + str(i)] = transformed_labels[:, i]

Class_dict

{'crescent_gap': 1, 
'crease': 2, 
'silk_spot': 3, 
'water_spot': 4, 
'welding_line': 5, 
'inclusion': 6, 
'oil_spot': 7, 
'waist folding': 8, 
'rolled_pit': 9, 
'punching_hole': 10}

PRE-PROCESSING

In this section the tasks done are as follows:

• Analyzing rectangular boxes in the XML files- that’s reading the file, extracting minimum and maximum x, y values
• Setting values to the labels
• Batch Normalization
• Shuffling of the batches
• Splitting dataset for training and validation
• Check from the training data

def to_labels(path):
    xml = open('{}'.format(path)).read()                         
    sel = etree.HTML(xml)                     
    width = int(sel.xpath('//size/width/text()')[0])    
    height = int(sel.xpath('//size/height/text()')[0])    
    xmin = int(sel.xpath('//bndbox/xmin/text()')[0])
    xmax = int(sel.xpath('//bndbox/xmax/text()')[0])
    ymin = int(sel.xpath('//bndbox/ymin/text()')[0])
    ymax = int(sel.xpath('//bndbox/ymax/text()')[0])
    return [xmin/width, ymin/height, xmax/width, ymax/height] 
labels = [to_labels(path) for path in xmls_path]
out1,out2,out3,out4 = list(zip(*labels))        

out1 = np.array(out1)
out2 = np.array(out2)
out3 = np.array(out3)
out4 = np.array(out4)
label = np.array(names.values)

label_datasets = tf.data.Dataset.from_tensor_slices((out1,out2,out3,out4,label))

def load_image(path):
    image = tf.io.read_file(path)                           
    image = tf.image.decode_jpeg(image,3)               
    image = tf.image.resize(image,[224,224])               
    image = tf.cast(image/127.5-1,tf.float32)                 
    return image    
BATCH_SIZE = 16
AUTO = tf.data.experimental.AUTOTUNE
dataset_label = dataset_label.repeat().shuffle(500).batch(BATCH_SIZE)
dataset_label = dataset_label.prefetch(AUTO)
test_count = int(len(imgs_train)*0.2)
train_count = len(imgs_train) - test_count
train_dataset = dataset_label.skip(test_count)
test_dataset = dataset_label.take(test_count)
species_dict = {v:k for k,v in Class_dict.items()}
for img, label in train_dataset.take(1):
    plt.imshow(keras.preprocessing.image.array_to_img(img[0]))     
    out1,out2,out3,out4,out5 = label                            
    xmin,ymin,xmax,ymax = out1[0].numpy()*224,out2[0].numpy()*224,out3[0].numpy()*224,out4[0].numpy()*224
    rect = Rectangle((xmin,ymin),(xmax-xmin),(ymax-ymin),fill=False,color='r')  
    ax = plt.gca()                      
    ax.axes.add_patch(rect)   
    pred_imglist = []
    pred_imglist.append(species_dict[np.argmax(out5[0])+1])
    plt.title(pred_imglist)
    plt.show()

BUILD NETWORK

In this section, the previous section weights of the Xception network are procured. Then, few simple layers are built over the exception network including dense, dropout layers for avoiding overfitting and increasing the acc of the training data. Also, declaring four output layers to analyze the results at different points. Displaying the summary of the built model using summary().

conv = keras.applications.xception.Xception(weights='imagenet',include_top=False, input_shape=(224,224,3), pooling='avg')
conv.trainable = True
inputs = keras.Input(shape=(224,224,3))
x = conv(inputs)
x1 = keras.layers.Dense(1024,activation='relu')(x)
x1 = keras.layers.Dense(512,activation='relu')(x1)


out1 = keras.layers.Dense(1,name='out1')(x1)
out2 = keras.layers.Dense(1,name='out2')(x1)
out3 = keras.layers.Dense(1,name='out3')(x1)
out4 = keras.layers.Dense(1,name='out4')(x1)

x2 = keras.layers.Dense(1024,activation='relu')(x)
x2 = keras.layers.Dropout(0.5)(x2)
x2 = keras.layers.Dense(512,activation='relu')(x2)
out_class = keras.layers.Dense(10,activation='softmax',name='out_item')(x2)

out = [out1,out2,out3,out4,out_class]

model = keras.models.Model(inputs=inputs,outputs=out)
model.summary()

Downloading data 83689472/83683744 [==============================] - 1s 0us/step
Model: "model" 
__________________________________________________________________________________________________ 
Layer (type)                Output Shape             Param #             Connected to 
================================================================================================== 
input_2 (InputLayer)    [(None, 224, 224, 3)            0 
__________________________________________________________________________________________________ 
xception (Model)            (None, 2048)             20861480           input_2[0][0] 
__________________________________________________________________________________________________ 
dense_2 (Dense)             (None, 1024)              2098176           xception[1][0] 
__________________________________________________________________________________________________ 
dense (Dense)               (None, 1024)              2098176           xception[1][0] 
__________________________________________________________________________________________________ 
dropout (Dropout)           (None, 1024)                 0              dense_2[0][0] 
__________________________________________________________________________________________________ 
dense_1 (Dense)             (None, 512)               524800            dense[0][0] 
__________________________________________________________________________________________________ 
dense_3 (Dense)             (None, 512)               524800            dropout[0][0] 
__________________________________________________________________________________________________ 
out1 (Dense)                 (None, 1)                 513              dense_1[0][0] 
__________________________________________________________________________________________________ 
out2 (Dense)                 (None, 1)                 513              dense_1[0][0] 
__________________________________________________________________________________________________ 
out3 (Dense)                 (None, 1)                 513              dense_1[0][0] 
__________________________________________________________________________________________________ 
out4 (Dense)                 (None, 1)                 513              dense_1[0][0] 
__________________________________________________________________________________________________ 
out_item (Dense)             (None, 10)                5130             dense_3[0][0] 
================================================================================================== 
Total params: 26,114,614 
Trainable params: 26,060,086 
Non-trainable params: 54,528 
__________________________________________________________________________________________________

CALLBACK AND COMPILATION

Declaration of callback functions. lr_reduce is the callback function used for training here. It will check for changes in the validation loss and if it remains the same or close to the same loss, then the training will stop thereby not wasting the computation. Then, a compilation of the network and made ready for training.

model.compile(keras.optimizers.Adam(0.0003), loss={'out1':'mse', 'out2':'mse', 'out3':'mse', 'out4':'mse', 'out_item':'categorical_crossentropy'}, metrics=['mae','acc'])
lr_reduce = keras.callbacks.ReduceLROnPlateau('val_loss', patience=6, factor=0.5, min_lr=1e-6)

TRAIN

Training of the model using fit() with 60 epochs and the lr_reduce callback function.

history = model.fit(train_dataset, steps_per_epoch=train_count//BATCH_SIZE, epochs=50, validation_data=test_dataset, validation_steps=test_count//BATCH_SIZE)

Epoch 1/20
166/166 [==============================] - 20s 120ms/step - loss: 1.6094 - accuracy: 0.3998 - val_loss: 1.1367 - val_accuracy: 0.6111
Epoch 2/20
166/166 [==============================] - 18s 106ms/step - loss: 1.0396 - accuracy: 0.6244 - val_loss: 0.4878 - val_accuracy: 0.8611
Epoch 3/20
166/166 [==============================] - 17s 104ms/step - loss: 0.7631 - accuracy: 0.7500 - val_loss: 0.4796 - val_accuracy: 0.8750
Epoch 4/20
166/166 [==============================] - 19s 112ms/step - loss: 0.6107 - accuracy: 0.8080 - val_loss: 0.1934 - val_accuracy: 0.9722
Epoch 5/20
166/166 [==============================] - 18s 106ms/step - loss: 0.5184 - accuracy: 0.8496 - val_loss: 0.1868 - val_accuracy: 0.9583
Epoch 6/20
166/166 [==============================] - 17s 104ms/step - loss: 0.4504 - accuracy: 0.8659 - val_loss: 0.1092 - val_accuracy: 0.9722
Epoch 7/20
166/166 [==============================] - 18s 108ms/step - loss: 0.4278 - accuracy: 0.8690 - val_loss: 0.2326 - val_accuracy: 0.9167
Epoch 8/20
166/166 [==============================] - 18s 106ms/step - loss: 0.3650 - accuracy: 0.8816 - val_loss: 0.1535 - val_accuracy: 0.9722
Epoch 9/20
166/166 [==============================] - 18s 106ms/step - loss: 0.3507 - accuracy: 0.8973 - val_loss: 0.3148 - val_accuracy: 0.9306
Epoch 10/20
166/166 [==============================] - 17s 103ms/step - loss: 0.3236 - accuracy: 0.8979 - val_loss: 0.2201 - val_accuracy: 0.9861
Epoch 11/20
166/166 [==============================] - 18s 111ms/step - loss: 0.3562 - accuracy: 0.9082 - val_loss: 0.2459 - val_accuracy: 0.9306
Epoch 12/20
166/166 [==============================] - 18s 107ms/step - loss: 0.3426 - accuracy: 0.8998 - val_loss: 0.3355 - val_accuracy: 0.9306
Epoch 13/20
166/166 [==============================] - 17s 102ms/step - loss: 0.2926 - accuracy: 0.9191 - val_loss: 0.1013 - val_accuracy: 0.9583
Epoch 14/20
166/166 [==============================] - 19s 113ms/step - loss: 0.3139 - accuracy: 0.9155 - val_loss: 0.0457 - val_accuracy: 1.0000
Epoch 15/20
166/166 [==============================] - 17s 104ms/step - loss: 0.2628 - accuracy: 0.9197 - val_loss: 0.0244 - val_accuracy: 1.0000
Epoch 16/20
166/166 [==============================] - 18s 107ms/step - loss: 0.2708 - accuracy: 0.9245 - val_loss: 0.6448 - val_accuracy: 0.9306
Epoch 17/20
166/166 [==============================] - 17s 105ms/step - loss: 0.2454 - accuracy: 0.9269 - val_loss: 0.0711 - val_accuracy: 0.9722
Epoch 18/20
166/166 [==============================] - 18s 110ms/step - loss: 0.2465 - accuracy: 0.9360 - val_loss: 0.0268 - val_accuracy: 1.0000
Epoch 19/20
166/166 [==============================] - 17s 104ms/step - loss: 0.2469 - accuracy: 0.9342 - val_loss: 0.0213 - val_accuracy: 1.0000
Epoch 20/20
166/166 [==============================] - 17s 103ms/step - loss: 0.2531 - accuracy: 0.9269 - val_loss: 0.0351 - val_accuracy: 0.9861

PLOT

Plotting of the data obtained while training. Thus the plots shown below are:

• Mean Absolute Error (MAE) of the validation set vs Epoch
• Mean Square Error (MSE) vs Epoch
• Accuracy of the validation set vs Epoch

def plot_history(history):                
    hist = pd.DataFrame(history.history)           
    hist['epoch']=history.epoch
    
    plt.figure()                                     
    plt.xlabel('Epoch')
    plt.ylabel('MSE')               
    plt.plot(hist['epoch'],hist['loss'],
            label='Train Loss')
    plt.plot(hist['epoch'],hist['val_loss'],
            label='Val Loss')                           
    plt.legend()
    
    plt.figure()                                      
    plt.xlabel('Epoch')
    plt.ylabel('Val_MAE')               
    plt.plot(hist['epoch'],hist['val_out1_mae'],
            label='Out1_mae')
    plt.plot(hist['epoch'],hist['val_out2_mae'],
            label='Out2_mae')
    plt.plot(hist['epoch'],hist['val_out3_mae'],
            label='Out3_mae')
    plt.plot(hist['epoch'],hist['val_out4_mae'],
            label='Out4_mae')
    plt.legend()      
    
    plt.figure()                                      
    plt.xlabel('Epoch')
    plt.ylabel('Val_Item_Acc')               
    plt.plot(hist['epoch'],hist['val_out_item_acc'],
            label='Out5_acc')
    
    plt.show()
    
plot_history(history)

EVALUATION

The evaluation of the trained model. Also, the 4 output layers are evaluated.

mae = model.evaluate(test_dataset)
print('out1_mae in test:{}'.format(mae[6]))
print('out2_mae in test:{}'.format(mae[8]))
print('out3_mae in test:{}'.format(mae[10]))
print('out4_mae in test:{}'.format(mae[12]))
print('class_label in test:{}'.format(mae[15]))

456/456 [==============================] - 79s 174ms/step - loss: 0.0737 - out1_loss: 0.0067 - out2_loss: 0.0088 - out3_loss: 0.0083 - out4_loss: 0.0096 - out_item_loss: 0.0403 - out1_mae: 0.0538 - out1_acc: 0.0000e+00 - out2_mae: 0.0650 - out2_acc: 0.0000e+00 - out3_mae: 0.0623 - out3_acc: 0.0925 - out4_mae: 0.0700 - out4_acc: 0.0798 - out_item_mae: 0.0033 - out_item_acc: 0.9863

out1_mae in test:0.05383157730102539
out2_mae in test:0.0649830773472786
out3_mae in test:0.06232777237892151
out4_mae in test:0.06997726112604141
class_label in test:0.9862938523292542

TESTING

Testing of the trained model and dimensional analysis of the defect using bounding boxes. Thus, we will know the exact location of the defect.

plt.figure(figsize=(10,24))
for img,_ in train_dataset.take(1):
    out1,out2,out3,out4,label = model.predict(img)
    for i in range(3):
        plt.subplot(3,1,i+1)            
        plt.imshow(keras.preprocessing.image.array_to_img(img[i]))    
        pred_imglist = []
        pred_imglist.append(species_dict[np.argmax(out5[i])+1])
        plt.title(pred_imglist)
        xmin,ymin,xmax,ymax = out1[i]*224,out2[i]*224,out3[i]*224,out4[i]*224
        rect = Rectangle((xmin,ymin),(xmax-xmin),(ymax-ymin),fill=False,color='r') 
        ax = plt.gca()                   
        ax.axes.add_patch(rect)

FINAL THOUGHTS

In this article, we thus looked into steel defect detection using the Xception network resulting in a validation accuracy of 98%  with just 15 epochs. Thus the inspection systems very stable and robust for usage. The architecture of the system is as follows: Firstly, procuring of the dataset, followed by pre-processing and loading the data. Then, the generation of the data and building the layers of the network followed by compiling and training of the network. Then, evaluation and plotting of the stats observed while training the model.

The source code for the steel surface inspection can be found and downloaded from here.

To learn transfer learning approaches, you can refer to my blogs:

[1]. Image classification of Bird species (Used a VGG16 net)

[2]. Salt Identification (Used UNets fused with a Residual net)

[3]. Fruit Species detection (Used Squeeze-Excitation Networks)

To learn from my other machine learning blogs, refer here.

REFERENCES:

[1]. Xcpetion: Deep Learning with Depthwise Separable Convolutions

Thank you. Hope this article was helpful for all!