Steel Surface Inspection in Keras | Python
Steel surface inspection is usually done by professionals and it is a very important inspection especially for social infrastructures. Large-scale inspection and position capturing by professionals in high acc are practically not possible. Moreover, it will be very expensive to rent such professionals. Thus, robots for such inspection systems increase the acc of the system and its consistency.
In this article, we are going to do the task of autonomous steel surface inspection using Xception Network is experimented with and analyzed in TensorFlow Keras API with complete Python code. A result of 98.61% in validation accuracy is obtained as a result of the built network. The workflow of this coming sections is as follows:
- Importing libraries
- Loading data
- Pre-processing the loaded data
- Building the network from the Xception Net
- Declaring the callback functions
- Compilation and Training
- Plotting for visualization
Happy Reading!!
LIBRARIES
Importing Keras TensorFlow and all the other required Python libraries.
import tensorflow as tf from tensorflow import keras import matplotlib.pyplot as plt import numpy as np import pandas as pd import glob import random from matplotlib.patches import Rectangle from lxml import etree %matplotlib inline
LOAD DATA
In this section the tasks done are as follows:
- Storing all the images in a path file
- Extraction of all the XML files
- Extraction of all the Image files
- Checking the image to labels
- Storing the label names separately
image_path = glob.glob('../input/images/images/*/*.jpg') xmls_path = glob.glob('../input/label/label/*.xml') xmls_train = [p.split('/')[-1].split('.')[0] for p in xmls_path] imgs_train = [img for img in image_path if (img.split('/')[-1].split)('.jpg')[0] in xmls_train] xmls_path.sort(key=lambda x:x.split('/')[-1].split('.xml')[0]) imgs_train.sort(key=lambda x:x.split('/')[-1].split('.jpg')[0]) names = [x.split("/")[-2] for x in imgs_train] names = pd.DataFrame(names,columns=['Types']) from sklearn.preprocessing import LabelBinarizer Class = names['Types'].unique() Class_dict = dict(zip(Class, range(1,len(Class)+1))) names['str'] = names['Types'].apply(lambda x: Class_dict[x]) lb = LabelBinarizer() lb.fit(list(Class_dict.values())) transformed_labels = lb.transform(names['str']) y_bin_labels = [] for i in range(transformed_labels.shape[1]): y_bin_labels.append('str' + str(i)) names['str' + str(i)] = transformed_labels[:, i] Class_dict
{'crescent_gap': 1, 'crease': 2, 'silk_spot': 3, 'water_spot': 4, 'welding_line': 5, 'inclusion': 6, 'oil_spot': 7, 'waist folding': 8, 'rolled_pit': 9, 'punching_hole': 10}
PRE-PROCESSING
In this section the tasks done are as follows:
- Analyzing rectangular boxes in the XML files- that’s reading the file, extracting minimum and maximum x, y values
- Setting values to the labels
- Batch Normalization
- Shuffling of the batches
- Splitting dataset for training and validation
- Check from the training data
def to_labels(path): xml = open('{}'.format(path)).read() sel = etree.HTML(xml) width = int(sel.xpath('//size/width/text()')[0]) height = int(sel.xpath('//size/height/text()')[0]) xmin = int(sel.xpath('//bndbox/xmin/text()')[0]) xmax = int(sel.xpath('//bndbox/xmax/text()')[0]) ymin = int(sel.xpath('//bndbox/ymin/text()')[0]) ymax = int(sel.xpath('//bndbox/ymax/text()')[0]) return [xmin/width, ymin/height, xmax/width, ymax/height] labels = [to_labels(path) for path in xmls_path] out1,out2,out3,out4 = list(zip(*labels)) out1 = np.array(out1) out2 = np.array(out2) out3 = np.array(out3) out4 = np.array(out4) label = np.array(names.values) label_datasets = tf.data.Dataset.from_tensor_slices((out1,out2,out3,out4,label)) def load_image(path): image = tf.io.read_file(path) image = tf.image.decode_jpeg(image,3) image = tf.image.resize(image,[224,224]) image = tf.cast(image/127.5-1,tf.float32) return image BATCH_SIZE = 16 AUTO = tf.data.experimental.AUTOTUNE dataset_label = dataset_label.repeat().shuffle(500).batch(BATCH_SIZE) dataset_label = dataset_label.prefetch(AUTO) test_count = int(len(imgs_train)*0.2) train_count = len(imgs_train) - test_count train_dataset = dataset_label.skip(test_count) test_dataset = dataset_label.take(test_count) species_dict = {v:k for k,v in Class_dict.items()} for img, label in train_dataset.take(1): plt.imshow(keras.preprocessing.image.array_to_img(img[0])) out1,out2,out3,out4,out5 = label xmin,ymin,xmax,ymax = out1[0].numpy()*224,out2[0].numpy()*224,out3[0].numpy()*224,out4[0].numpy()*224 rect = Rectangle((xmin,ymin),(xmax-xmin),(ymax-ymin),fill=False,color='r') ax = plt.gca() ax.axes.add_patch(rect) pred_imglist = [] pred_imglist.append(species_dict[np.argmax(out5[0])+1]) plt.title(pred_imglist) plt.show()
BUILD NETWORK
In this section, the previous section weights of the Xception network are procured. Then, few simple layers are built over the exception network including dense, dropout layers for avoiding overfitting and increasing the acc of the training data. Also, declaring four output layers to analyze the results at different points. Displaying the summary of the built model using summary().
conv = keras.applications.xception.Xception(weights='imagenet',include_top=False, input_shape=(224,224,3), pooling='avg') conv.trainable = True inputs = keras.Input(shape=(224,224,3)) x = conv(inputs) x1 = keras.layers.Dense(1024,activation='relu')(x) x1 = keras.layers.Dense(512,activation='relu')(x1) out1 = keras.layers.Dense(1,name='out1')(x1) out2 = keras.layers.Dense(1,name='out2')(x1) out3 = keras.layers.Dense(1,name='out3')(x1) out4 = keras.layers.Dense(1,name='out4')(x1) x2 = keras.layers.Dense(1024,activation='relu')(x) x2 = keras.layers.Dropout(0.5)(x2) x2 = keras.layers.Dense(512,activation='relu')(x2) out_class = keras.layers.Dense(10,activation='softmax',name='out_item')(x2) out = [out1,out2,out3,out4,out_class] model = keras.models.Model(inputs=inputs,outputs=out) model.summary()
Downloading data 83689472/83683744 [==============================] - 1s 0us/step Model: "model" __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_2 (InputLayer) [(None, 224, 224, 3) 0 __________________________________________________________________________________________________ xception (Model) (None, 2048) 20861480 input_2[0][0] __________________________________________________________________________________________________ dense_2 (Dense) (None, 1024) 2098176 xception[1][0] __________________________________________________________________________________________________ dense (Dense) (None, 1024) 2098176 xception[1][0] __________________________________________________________________________________________________ dropout (Dropout) (None, 1024) 0 dense_2[0][0] __________________________________________________________________________________________________ dense_1 (Dense) (None, 512) 524800 dense[0][0] __________________________________________________________________________________________________ dense_3 (Dense) (None, 512) 524800 dropout[0][0] __________________________________________________________________________________________________ out1 (Dense) (None, 1) 513 dense_1[0][0] __________________________________________________________________________________________________ out2 (Dense) (None, 1) 513 dense_1[0][0] __________________________________________________________________________________________________ out3 (Dense) (None, 1) 513 dense_1[0][0] __________________________________________________________________________________________________ out4 (Dense) (None, 1) 513 dense_1[0][0] __________________________________________________________________________________________________ out_item (Dense) (None, 10) 5130 dense_3[0][0] ================================================================================================== Total params: 26,114,614 Trainable params: 26,060,086 Non-trainable params: 54,528 __________________________________________________________________________________________________
CALLBACK AND COMPILATION
Declaration of callback functions. lr_reduce is the callback function used for training here. It will check for changes in the validation loss and if it remains the same or close to the same loss, then the training will stop thereby not wasting the computation. Then, a compilation of the network and made ready for training.
model.compile(keras.optimizers.Adam(0.0003), loss={'out1':'mse', 'out2':'mse', 'out3':'mse', 'out4':'mse', 'out_item':'categorical_crossentropy'}, metrics=['mae','acc']) lr_reduce = keras.callbacks.ReduceLROnPlateau('val_loss', patience=6, factor=0.5, min_lr=1e-6)
TRAIN
Training of the model using fit() with 60 epochs and the lr_reduce callback function.
history = model.fit(train_dataset, steps_per_epoch=train_count//BATCH_SIZE, epochs=50, validation_data=test_dataset, validation_steps=test_count//BATCH_SIZE)
Epoch 1/20 166/166 [==============================] - 20s 120ms/step - loss: 1.6094 - accuracy: 0.3998 - val_loss: 1.1367 - val_accuracy: 0.6111 Epoch 2/20 166/166 [==============================] - 18s 106ms/step - loss: 1.0396 - accuracy: 0.6244 - val_loss: 0.4878 - val_accuracy: 0.8611 Epoch 3/20 166/166 [==============================] - 17s 104ms/step - loss: 0.7631 - accuracy: 0.7500 - val_loss: 0.4796 - val_accuracy: 0.8750 Epoch 4/20 166/166 [==============================] - 19s 112ms/step - loss: 0.6107 - accuracy: 0.8080 - val_loss: 0.1934 - val_accuracy: 0.9722 Epoch 5/20 166/166 [==============================] - 18s 106ms/step - loss: 0.5184 - accuracy: 0.8496 - val_loss: 0.1868 - val_accuracy: 0.9583 Epoch 6/20 166/166 [==============================] - 17s 104ms/step - loss: 0.4504 - accuracy: 0.8659 - val_loss: 0.1092 - val_accuracy: 0.9722 Epoch 7/20 166/166 [==============================] - 18s 108ms/step - loss: 0.4278 - accuracy: 0.8690 - val_loss: 0.2326 - val_accuracy: 0.9167 Epoch 8/20 166/166 [==============================] - 18s 106ms/step - loss: 0.3650 - accuracy: 0.8816 - val_loss: 0.1535 - val_accuracy: 0.9722 Epoch 9/20 166/166 [==============================] - 18s 106ms/step - loss: 0.3507 - accuracy: 0.8973 - val_loss: 0.3148 - val_accuracy: 0.9306 Epoch 10/20 166/166 [==============================] - 17s 103ms/step - loss: 0.3236 - accuracy: 0.8979 - val_loss: 0.2201 - val_accuracy: 0.9861 Epoch 11/20 166/166 [==============================] - 18s 111ms/step - loss: 0.3562 - accuracy: 0.9082 - val_loss: 0.2459 - val_accuracy: 0.9306 Epoch 12/20 166/166 [==============================] - 18s 107ms/step - loss: 0.3426 - accuracy: 0.8998 - val_loss: 0.3355 - val_accuracy: 0.9306 Epoch 13/20 166/166 [==============================] - 17s 102ms/step - loss: 0.2926 - accuracy: 0.9191 - val_loss: 0.1013 - val_accuracy: 0.9583 Epoch 14/20 166/166 [==============================] - 19s 113ms/step - loss: 0.3139 - accuracy: 0.9155 - val_loss: 0.0457 - val_accuracy: 1.0000 Epoch 15/20 166/166 [==============================] - 17s 104ms/step - loss: 0.2628 - accuracy: 0.9197 - val_loss: 0.0244 - val_accuracy: 1.0000 Epoch 16/20 166/166 [==============================] - 18s 107ms/step - loss: 0.2708 - accuracy: 0.9245 - val_loss: 0.6448 - val_accuracy: 0.9306 Epoch 17/20 166/166 [==============================] - 17s 105ms/step - loss: 0.2454 - accuracy: 0.9269 - val_loss: 0.0711 - val_accuracy: 0.9722 Epoch 18/20 166/166 [==============================] - 18s 110ms/step - loss: 0.2465 - accuracy: 0.9360 - val_loss: 0.0268 - val_accuracy: 1.0000 Epoch 19/20 166/166 [==============================] - 17s 104ms/step - loss: 0.2469 - accuracy: 0.9342 - val_loss: 0.0213 - val_accuracy: 1.0000 Epoch 20/20 166/166 [==============================] - 17s 103ms/step - loss: 0.2531 - accuracy: 0.9269 - val_loss: 0.0351 - val_accuracy: 0.9861
PLOT
Plotting of the data obtained while training. Thus the plots shown below are:
- Mean Absolute Error (MAE) of the validation set vs Epoch
- Mean Square Error (MSE) vs Epoch
- Accuracy of the validation set vs Epoch
def plot_history(history): hist = pd.DataFrame(history.history) hist['epoch']=history.epoch plt.figure() plt.xlabel('Epoch') plt.ylabel('MSE') plt.plot(hist['epoch'],hist['loss'], label='Train Loss') plt.plot(hist['epoch'],hist['val_loss'], label='Val Loss') plt.legend() plt.figure() plt.xlabel('Epoch') plt.ylabel('Val_MAE') plt.plot(hist['epoch'],hist['val_out1_mae'], label='Out1_mae') plt.plot(hist['epoch'],hist['val_out2_mae'], label='Out2_mae') plt.plot(hist['epoch'],hist['val_out3_mae'], label='Out3_mae') plt.plot(hist['epoch'],hist['val_out4_mae'], label='Out4_mae') plt.legend() plt.figure() plt.xlabel('Epoch') plt.ylabel('Val_Item_Acc') plt.plot(hist['epoch'],hist['val_out_item_acc'], label='Out5_acc') plt.show() plot_history(history)
EVALUATION
The evaluation of the trained model. Also, the 4 output layers are evaluated.
mae = model.evaluate(test_dataset) print('out1_mae in test:{}'.format(mae[6])) print('out2_mae in test:{}'.format(mae[8])) print('out3_mae in test:{}'.format(mae[10])) print('out4_mae in test:{}'.format(mae[12])) print('class_label in test:{}'.format(mae[15]))
456/456 [==============================] - 79s 174ms/step - loss: 0.0737 - out1_loss: 0.0067 - out2_loss: 0.0088 - out3_loss: 0.0083 - out4_loss: 0.0096 - out_item_loss: 0.0403 - out1_mae: 0.0538 - out1_acc: 0.0000e+00 - out2_mae: 0.0650 - out2_acc: 0.0000e+00 - out3_mae: 0.0623 - out3_acc: 0.0925 - out4_mae: 0.0700 - out4_acc: 0.0798 - out_item_mae: 0.0033 - out_item_acc: 0.9863
out1_mae in test:0.05383157730102539 out2_mae in test:0.0649830773472786 out3_mae in test:0.06232777237892151 out4_mae in test:0.06997726112604141 class_label in test:0.9862938523292542
TESTING
Testing of the trained model and dimensional analysis of the defect using bounding boxes. Thus, we will know the exact location of the defect.
plt.figure(figsize=(10,24)) for img,_ in train_dataset.take(1): out1,out2,out3,out4,label = model.predict(img) for i in range(3): plt.subplot(3,1,i+1) plt.imshow(keras.preprocessing.image.array_to_img(img[i])) pred_imglist = [] pred_imglist.append(species_dict[np.argmax(out5[i])+1]) plt.title(pred_imglist) xmin,ymin,xmax,ymax = out1[i]*224,out2[i]*224,out3[i]*224,out4[i]*224 rect = Rectangle((xmin,ymin),(xmax-xmin),(ymax-ymin),fill=False,color='r') ax = plt.gca() ax.axes.add_patch(rect)
FINAL THOUGHTS
In this article, we thus looked into steel defect detection using the Xception network resulting in a validation accuracy of 98% with just 15 epochs. Thus the inspection systems very stable and robust for usage. The architecture of the system is as follows: Firstly, procuring of the dataset, followed by pre-processing and loading the data. Then, the generation of the data and building the layers of the network followed by compiling and training of the network. Then, evaluation and plotting of the stats observed while training the model.
The source code for the steel surface inspection can be found and downloaded from here.
To learn transfer learning approaches, you can refer to my blogs:
[1]. Image classification of Bird species (Used a VGG16 net)
[2]. Salt Identification (Used UNets fused with a Residual net)
[3]. Fruit Species detection (Used Squeeze-Excitation Networks)
To learn from my other machine learning blogs, refer here.
REFERENCES:
[1]. Xcpetion: Deep Learning with Depthwise Separable Convolutions
Thank you. Hope this article was helpful for all!
I have received this error below in this code. Any comment about it? thanks
raise ValueError(‘y has 0 samples: %r’ % y)
ValueError: y has 0 samples: []