Lane Detection using Neural Networks with Keras
In this article, lane detection for autonomous vehicles with the help of convolutional neural networks is done. We have previously worked on Lane detection using Canny edge and Hough lines. But they are not the best method/ technique that can be used for autonomous lane detection. For good performance of the canny detectors, we will have to mask the desired region of interest (ROI) which varies from one vehicle to another. Moreover, those filters and edge detectors tend to perform relatively poorly in high steep areas. Also, with real-life observations using monocular cameras shadows, glares and rapid movement of the vehicle tend to result in poor frame clarity resulting in inaccurate detection. Thus, the introduction of neural networks for lane detection tends to solve the above-mentioned issues and thereby increasing the robustness and accuracy of the system.
The pipeline of the architecture reflected in this project are:
- Data-Preprocessing
- Building Convolutional Neural Net
- Training Model
- Saving Model
- Prediction in real-time
The article consists of 2 Python files and thus two sections for the ease of understanding of the user. The first section/ python script will train the dataset with the help of neural networks and save the trained model in the database. The second python file will implement the detection of lines in the input image/ video. We will use video in this article, as we will get video from monocular cameras in real-time for detection.
For learning about autonomous lane detection using traditional canny edge and hough lines, click here. Happy Reading!!!
SECTION I
In this section, initializing the dataset, creating the model, training, and saving the trained model (to be used in SECTION II) is done.
LIBRARIES
Importing necessary libraries for the project.
import numpy as np import pickle from sklearn.utils import shuffle from sklearn.model_selection import train_test_split from keras.models import Sequential from keras.layers import Activation, Dropout, UpSampling2D from keras.layers import Conv2DTranspose, Conv2D, MaxPooling2D from keras.layers.normalization import BatchNormalization from keras.preprocessing.image import ImageDataGenerator from keras import regularizers
CREATE LAYERS
Creating the neural network here. Some of the major used layers are Batch Normalization for normalizing the input images, convolutional layers, and max-pooling layers. Also, upsampling and deconvolution is added in between the layers for the efficiency of the model to be attained. The final layer will have only one channel targeting the lanes to be detected alone. Thus, the filter is set to be one.
Below is our Python code:
def create_model(input_shape, pool_size): model = Sequential() model.add(BatchNormalization(input_shape=input_shape)) model.add(Conv2D(8, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv1')) model.add(Conv2D(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv2')) model.add(MaxPooling2D(pool_size=pool_size)) model.add(Conv2D(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv3')) model.add(Dropout(0.2)) model.add(Conv2D(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv4')) model.add(Dropout(0.2)) model.add(Conv2D(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv5')) model.add(Dropout(0.2)) model.add(MaxPooling2D(pool_size=pool_size)) model.add(Conv2D(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv6')) model.add(Dropout(0.2)) model.add(Conv2D(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv7')) model.add(Dropout(0.2)) model.add(MaxPooling2D(pool_size=pool_size)) model.add(UpSampling2D(size=pool_size)) model.add(Conv2DTranspose(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv1')) model.add(Dropout(0.2)) model.add(Conv2DTranspose(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv2')) model.add(Dropout(0.2)) model.add(UpSampling2D(size=pool_size)) model.add(Conv2DTranspose(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv3')) model.add(Dropout(0.2)) model.add(Conv2DTranspose(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv4')) model.add(Dropout(0.2)) model.add(Conv2DTranspose(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv5')) model.add(Dropout(0.2)) model.add(UpSampling2D(size=pool_size)) model.add(Conv2DTranspose(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv6')) model.add(Conv2DTranspose(1, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Final')) return model
LOADING DATASET
The importing of the training images and the training labels into the notebook. Conversion of the inputted dataset into arrays for ease of usage with the neural networking.
def main(): train_images = pickle.load(open("full_CNN_train.p", "rb" )) labels = pickle.load(open("full_CNN_labels.p", "rb" )) train_images = np.array(train_images) labels = np.array(labels)
NORMALIZATION
Normalizing of the dataset for accurate and fair training. Then, shuffling and splitting of the dataset for training purposes into training and validation sets. The setting of the test set to be 10% for evaluation purposes.
labels = labels / 255 train_images, labels = shuffle(train_images, labels) X_train, X_val, y_train, y_val = train_test_split(train_images, labels, test_size=0.1)
PARAMETRICS
Allocation of the batch size, number of epochs, pooling size, and input shape.
batch_size = 128 epochs = 10 pool_size = (2, 2) input_shape = X_train.shape[1:]
CREATE MODEL
Creation of the model. Generators are used to limit the additional usage of the data thereby increasing the computation.
model = create_model(input_shape, pool_size) datagen = ImageDataGenerator(channel_shift_range=0.2) datagen.fit(X_train)
FIT GENERATOR
Architecting the data generation flow for the generator for training the model.
model.fit_generator(datagen.flow(X_train, y_train, batch_size=batch_size), steps_per_epoch=len(X_train)/batch_size, epochs=epochs, verbose=1, validation_data=(X_val, y_val))
TRAINING
Compiling and training of the model. Freezing of the previous layers by making the trainable layers False. Saving of the architecture and the weights of the model for the project into the database.
model.trainable = False model.compile(optimizer='Adam', loss='mean_squared_error') model.save('full_CNN_model.h5') model.summary()
IMPLEMENTATION
The main() acts as a way to start all the classes mentioned in SECTION I sequentially.
if __name__ == '__main__': main()
SECTION II
In this section, we will load the trained and saved model from the database, and implement it in real-time for lane detection using the monocular camera.
LIBRARIES
Importing necessary libraries for the project.
import numpy as np import cv2,glob,os import matplotlib.image as mpimg from scipy.misc import imresize from moviepy.editor import VideoFileClip from IPython.display import HTML from keras.models import model_from_json import matplotlib.pyplot as plt
DISPLAY TEST IMAGES
Created a class for plotting the images when required. Then, show_images() displays the lanes detected in the images or the sample images.
def show_images(images, cmap=None): cols = 2 rows = (len(images)+1)//cols plt.figure(figsize=(10, 11)) for i, image in enumerate(images): plt.subplot(rows, cols, i+1) cmap = 'gray' if len(image.shape)==2 else cmap plt.imshow(image, cmap=cmap) plt.xticks([]) plt.yticks([]) plt.tight_layout(pad=0, h_pad=0, w_pad=0) plt.show() test_images = [plt.imread(path) for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg')] show_images(test_images)
PRE-TRAINED MODEL
Loading and saving the model and its weights attained from SECTION I.
json_file = open('/content/drive/MyDrive/Approach-2/model.json', 'r') json_model = json_file.read() json_file.close() model = model_from_json(json_model) model.load_weights('/content/drive/MyDrive/Approach-2/model.h5')
LANE CLASS
Created a class called lane(). It averages the lanes.
class Lanes(): def __init__(self): self.recent_fit = [] self.avg_fit = []
PRE-PROCESSING IMAGE
Here, the road_lines_image() class normalizes all the input images along with the prediction of lanes with the neural network. Then, the calculations of the average detections and the merging of the detections with the original image for effectual visualization.
def road_lines_image(image): img_arr = cv2.imread(image) actual_image = imresize(img_arr, (720, 1280, 3)) img = mpimg.imread(image) small_img_2 = imresize(img, (80, 160, 3)) small_img_1= np.array(small_img_2) small_img = small_img_1[None, :, :, :] prediction = model.predict(small_img)[0] * 255 lanes.recent_fit.append(prediction) if len(lanes.recent_fit) > 5: lanes.recent_fit = lanes.recent_fit[1:] lanes.avg_fit = np.mean(np.array([i for i in lanes.recent_fit]), axis = 0) blanks = np.zeros_like(lanes.avg_fit).astype(np.uint8) lane_drawn = np.dstack((blanks, lanes.avg_fit, blanks)) lane_image = imresize(lane_drawn, (720, 1280, 3)) result = cv2.addWeighted(actual_image, 1, lane_image, 1, 0) return result
PREDICTION
In this part of the section, the works done are :
- Predicting the lane detector,
- creating the lane object, and
- saving the image.
lanes = Lanes() for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg'): res_img = road_lines_image(path) names = [os.path.basename(x) for x in glob.glob(path)] out_path = '/content/drive/MyDrive/Approach-2/test_images/'+names[0] cv2.imwrite(out_path,res_img)
PLOTTING
Plotting the prediction results for visualization.
predicted_images = [plt.imread(path) for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg')] show_images(predicted_images)
FINAL THOUGHTS
In this article, we thus discussed and expanded the scope of neural networks for autonomous lane detection. It has countless real-life applications especially in urban areas to be specific. Due to the high traffic in countries like India, there is a high possibility for the occurrence of accidents due to driver drowsiness, drunken drive, etc., which can positively be controlled using autonomous lane detection techniques. Thus, these systems can positively control the error rate of the vehicle by keeping note of the lane lines in the road.
The article thus consisted of 2 sections: the first being the training of the model from the dataset and saving it, and the second being the section to implement the model into lanes thus enabling effective lane detection. Hope the article helps.
To learn more from my Machine Learning blogs, click here.
The source code of this article can be downloaded from here.
Leave a Reply