Lane Detection using Neural Networks with Keras

By Jerrin Bright

In this article, lane detection for autonomous vehicles with the help of convolutional neural networks is done. We have previously worked on Lane detection using Canny edge and Hough lines. But they are not the best method/ technique that can be used for autonomous lane detection. For good performance of the canny detectors, we will have to mask the desired region of interest (ROI) which varies from one vehicle to another. Moreover, those filters and edge detectors tend to perform relatively poorly in high steep areas. Also, with real-life observations using monocular cameras shadows, glares and rapid movement of the vehicle tend to result in poor frame clarity resulting in inaccurate detection. Thus, the introduction of neural networks for lane detection tends to solve the above-mentioned issues and thereby increasing the robustness and accuracy of the system.

The pipeline of the architecture reflected in this project are:

  1. Data-Preprocessing
  2. Building Convolutional Neural Net
  3. Training Model
  4. Saving Model
  5. Prediction in real-time

The article consists of 2 Python files and thus two sections for the ease of understanding of the user. The first section/ python script will train the dataset with the help of neural networks and save the trained model in the database. The second python file will implement the detection of lines in the input image/ video. We will use video in this article, as we will get video from monocular cameras in real-time for detection.

For learning about autonomous lane detection using traditional canny edge and hough lines, click here. Happy Reading!!!

SECTION I

In this section, initializing the dataset, creating the model, training, and saving the trained model (to be used in SECTION II) is done.

LIBRARIES

Importing necessary libraries for the project.

import numpy as np
import pickle
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Activation, Dropout, UpSampling2D
from keras.layers import Conv2DTranspose, Conv2D, MaxPooling2D
from keras.layers.normalization import BatchNormalization
from keras.preprocessing.image import ImageDataGenerator
from keras import regularizers

CREATE LAYERS

Creating the neural network here. Some of the major used layers are Batch Normalization for normalizing the input images, convolutional layers, and max-pooling layers. Also, upsampling and deconvolution is added in between the layers for the efficiency of the model to be attained. The final layer will have only one channel targeting the lanes to be detected alone. Thus, the filter is set to be one.

Below is our Python code:

def create_model(input_shape, pool_size):
    model = Sequential()
    model.add(BatchNormalization(input_shape=input_shape))
    model.add(Conv2D(8, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv1'))
    model.add(Conv2D(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv2'))
    model.add(MaxPooling2D(pool_size=pool_size))
    model.add(Conv2D(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv3'))
    model.add(Dropout(0.2))
    model.add(Conv2D(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv4'))
    model.add(Dropout(0.2))
    model.add(Conv2D(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv5'))
    model.add(Dropout(0.2))
    model.add(MaxPooling2D(pool_size=pool_size))
    model.add(Conv2D(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv6'))
    model.add(Dropout(0.2))
    model.add(Conv2D(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Conv7'))
    model.add(Dropout(0.2))
    model.add(MaxPooling2D(pool_size=pool_size))
    model.add(UpSampling2D(size=pool_size))
    model.add(Conv2DTranspose(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv1'))
    model.add(Dropout(0.2))
    model.add(Conv2DTranspose(64, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv2'))
    model.add(Dropout(0.2))
    model.add(UpSampling2D(size=pool_size))
    model.add(Conv2DTranspose(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv3'))
    model.add(Dropout(0.2))
    model.add(Conv2DTranspose(32, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv4'))
    model.add(Dropout(0.2))
    model.add(Conv2DTranspose(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv5'))
    model.add(Dropout(0.2))
    model.add(UpSampling2D(size=pool_size))
    model.add(Conv2DTranspose(16, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Deconv6'))
    model.add(Conv2DTranspose(1, (3, 3), padding='valid', strides=(1,1), activation = 'relu', name = 'Final'))
    return model

LOADING DATASET

The importing of the training images and the training labels into the notebook. Conversion of the inputted dataset into arrays for ease of usage with the neural networking.

def main():
    train_images = pickle.load(open("full_CNN_train.p", "rb" ))
    labels = pickle.load(open("full_CNN_labels.p", "rb" ))
    train_images = np.array(train_images)
    labels = np.array(labels)

NORMALIZATION

Normalizing of the dataset for accurate and fair training. Then, shuffling and splitting of the dataset for training purposes into training and validation sets. The setting of the test set to be 10% for evaluation purposes.

   labels = labels / 255
   train_images, labels = shuffle(train_images, labels)
   X_train, X_val, y_train, y_val = train_test_split(train_images, labels,       test_size=0.1)

PARAMETRICS

Allocation of the batch size, number of epochs, pooling size, and input shape.

   batch_size = 128
   epochs = 10
   pool_size = (2, 2)
   input_shape = X_train.shape[1:]

CREATE MODEL

Creation of the model. Generators are used to limit the additional usage of the data thereby increasing the computation.

   model = create_model(input_shape, pool_size)
   datagen = ImageDataGenerator(channel_shift_range=0.2)
   datagen.fit(X_train)

FIT GENERATOR

Architecting the data generation flow for the generator for training the model.

   model.fit_generator(datagen.flow(X_train, y_train, batch_size=batch_size), steps_per_epoch=len(X_train)/batch_size,
   epochs=epochs, verbose=1, validation_data=(X_val, y_val))

TRAINING

Compiling and training of the model. Freezing of the previous layers by making the trainable layers False. Saving of the architecture and the weights of the model for the project into the database.

   model.trainable = False
   model.compile(optimizer='Adam', loss='mean_squared_error')
   model.save('full_CNN_model.h5')
   model.summary()

IMPLEMENTATION

The main() acts as a way to start all the classes mentioned in SECTION I sequentially.

if __name__ == '__main__':
    main()

 

SECTION II

In this section, we will load the trained and saved model from the database, and implement it in real-time for lane detection using the monocular camera.

LIBRARIES

Importing necessary libraries for the project.

import numpy as np
import cv2,glob,os
import matplotlib.image as mpimg
from scipy.misc import imresize
from moviepy.editor import VideoFileClip
from IPython.display import HTML
from keras.models import model_from_json
import matplotlib.pyplot as plt

DISPLAY TEST IMAGES

Created a class for plotting the images when required. Then, show_images() displays the lanes detected in the images or the sample images.

def show_images(images, cmap=None):
    cols = 2
    rows = (len(images)+1)//cols
    
    plt.figure(figsize=(10, 11))
    for i, image in enumerate(images):
        plt.subplot(rows, cols, i+1)

        cmap = 'gray' if len(image.shape)==2 else cmap
        plt.imshow(image, cmap=cmap)
        plt.xticks([])
        plt.yticks([])
    plt.tight_layout(pad=0, h_pad=0, w_pad=0)
    plt.show()

test_images = [plt.imread(path) for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg')]

show_images(test_images)

PRE-TRAINED MODEL

Loading and saving the model and its weights attained from SECTION I.

json_file = open('/content/drive/MyDrive/Approach-2/model.json', 'r')
json_model = json_file.read()
json_file.close()

model = model_from_json(json_model)

model.load_weights('/content/drive/MyDrive/Approach-2/model.h5')

LANE CLASS

Created a class called lane(). It averages the lanes.

class Lanes():
    def __init__(self):
        self.recent_fit = []
        self.avg_fit = []

PRE-PROCESSING IMAGE

Here, the road_lines_image() class normalizes all the input images along with the prediction of lanes with the neural network. Then, the calculations of the average detections and the merging of the detections with the original image for effectual visualization.

def road_lines_image(image):
    
    img_arr = cv2.imread(image)
    actual_image = imresize(img_arr, (720, 1280, 3))

    img = mpimg.imread(image)
    small_img_2 = imresize(img, (80, 160, 3))
    small_img_1= np.array(small_img_2)
    small_img = small_img_1[None, :, :, :]

    prediction = model.predict(small_img)[0] * 255

    lanes.recent_fit.append(prediction)
    if len(lanes.recent_fit) > 5:
        lanes.recent_fit = lanes.recent_fit[1:]

    lanes.avg_fit = np.mean(np.array([i for i in lanes.recent_fit]), axis = 0)

    blanks = np.zeros_like(lanes.avg_fit).astype(np.uint8)
    lane_drawn = np.dstack((blanks, lanes.avg_fit, blanks))

    lane_image = imresize(lane_drawn, (720, 1280, 3))

    result = cv2.addWeighted(actual_image, 1, lane_image, 1, 0)

    return result

PREDICTION

In this part of the section, the works done are :

  • Predicting the lane detector,
  • creating the lane object, and
  • saving the image.
lanes = Lanes()

for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg'):
    res_img = road_lines_image(path)
    names = [os.path.basename(x) for x in glob.glob(path)]   
    out_path = '/content/drive/MyDrive/Approach-2/test_images/'+names[0]
    cv2.imwrite(out_path,res_img)

PLOTTING

Plotting the prediction results for visualization.

predicted_images = [plt.imread(path) for path in glob.glob('/content/drive/MyDrive/Approach-2/test_images/*.jpg')]

show_images(predicted_images)

 

FINAL THOUGHTS

In this article, we thus discussed and expanded the scope of neural networks for autonomous lane detection. It has countless real-life applications especially in urban areas to be specific. Due to the high traffic in countries like India, there is a high possibility for the occurrence of accidents due to driver drowsiness, drunken drive, etc., which can positively be controlled using autonomous lane detection techniques. Thus, these systems can positively control the error rate of the vehicle by keeping note of the lane lines in the road.

The article thus consisted of 2 sections: the first being the training of the model from the dataset and saving it, and the second being the section to implement the model into lanes thus enabling effective lane detection. Hope the article helps.

To learn more from my Machine Learning blogs, click here.

The source code of this article can be downloaded from here.

Leave a Reply

Your email address will not be published. Required fields are marked *