Semantic and Instance Segmentation with PixelLib

In this tutorial, we will learn how to do semantic and instance image segmentation with help of the PixelLib library.

Image Segmentation

Some computer vision challenges need a more in-depth comprehension of the visual information. These challenges may not be amenable to classification and object identification. Image Segmentation was created in response to a demand for a reliable method for resolving some difficult computer vision challenges.
A collection of pixel values makes up every image. Image segmentation segments the image at a pixel level. By separating a picture into various segments based on the classes given to each of the pixel values included in the image, a machine may more efficiently analyze it.
Image Segmentation is useful for tackling key computer vision issues because of the unique approach it employs. These are issues that need a great deal of research. Because of the unique approach used in Image Segmentation, it may be used to solve key computer vision challenges. These are issues that need precise information on the objects in an image, information that cannot be obtained simply by categorizing the entire image or giving bounding boxes for the items in the image. Image Segmentation has made significant contributions in the following areas:

For a better grasp of the broad landscape, autonomous automobiles need a good visual system.
Medical image segmentation is used to segment body sections to perform diagnostic testing.
Satellite images’ analysis.

There are two major types of Image Segmentation:

Semantic Segmentation: In semantic segmentation, colormaps are used to segment objects that are categorized with the same pixel values.
Instance Segmentation: Instance segmentation differs from semantic segmentation in that various color mappings are used to separate distinct instances of the same item.

PixelLib

It is a library designed to make Image Segmentation in real-world issues simple to implement. PixelLib is a versatile library that may be used in software solutions that need Image Segmentation.

Import Libraries

Let’s import all the required libraries for image segmentation.

import pixellib
from pixellib.semantic import semantic_segmentation 
import tensorflow as tf

Semantic Segmentation

Let’s create an instance of a class for performing semantic segmentation from pixellib and call the function to load the xception model trained on pascal voc.

This xception model is trained on the pascal voc dataset, a dataset with 20 object categories.

Download the Xception model from here.

The line of code that performs image segmentation is done in the pascalvoc’s color format. This function takes in two parameters:

path_to_image: the path to the image to be segemented.
path_to_output_image: the path to save the output image. The image will be saved in your current working directory.

Code

segment=semantic_segmentation()
segment.load_pascalvoc_model("deeplabv3_xception_tf_dim_ordering_tf_kernels.h5")
segment.segmentAsPascalvoc("test.jpg",output_image_name="output2.jpg")

Output

Original Image

Segmented Image

Your saved image with all the objects present is segmented.

You can obtain an image with segmentation overlay on the objects with a modified code below.

segment_image.segmentAsPascalvoc("test.jpg", output_image_name = "output3.jpg", overlay = True)

Output

Instance Segmentation

The Instance segmentation is based on the Masked R-CNN framework.

The Mask R_CNN model is trained on the Microsoft Coco dataset, a dataset with 80 common object categories.

Let’s perform instance segmentation which we imported earlier.

Code

segment_intimage = instance_segmentation()
segment_intimage.load_model("mask_rcnn_coco.h5")
segment_intimage.segmentImage("sample2.jpg", output_image_name = "image_new.jpg")

The above code downloads the RCNN model to perform instance segmentation

Download the RCNN model from here.

This code performs instance segmentation on an image takes two parameters:

path_to_image: The path to the image to be predicted by the model.
output_image_name: The path to save the segmentation result. It will be saved in your current working directory.