Grid Search for Hyperparameter tuning in SVM using scikit-learn

By Aryan Karnati

SVM Introduction

In this article, I will show you how to improve your SVM(Support Vector Machine) model’s accuracy or other metrics, using Grid Search using scikit-learn in Python.  SVM chooses a hyperplane(best region or boundary) to segregate the objects into distinct classes. The points of each class very close to the hyperplane are called support vectors. The distance between these vectors and hyperplane is called a margin. Working of an SVM when there are only 2 classes in a 2d plane is shown below. This is an example of a linear boundary SVM.

Linear SVM Classifier.

The SVM, as you know is a supervised machine learning algorithm that chooses the decision boundary by taking into consideration the following:

a)Increase the distance of the decision boundary from support vectors, also known as margin.

b)Minimise the number of misclassified items.

Hyperparameters in SVM

A machine learning algorithm requires certain hyperparameters that must be tuned before training. An optimal subset of these hyperparameters must be selected, which is called hyperparameter optimization. Grid-Search is a sci-kit learn package that provides for hyperparameter tuning. A grid search space is generated by taking the initial set of values given to each hyperparameter. Each cell in the grid is searched for the optimal solution.

There are two hyperparameters to be tuned on an SVM model: C and gamma.

C value: C value adds a penalty each time an item is misclassified. So, a low C value has more misclassified items. On the other hand, a large C value has a plane with a very little margin which leads to overfitting.

γ value: It is not always possible to classify data using a linear kernel when data points are complexly arranged. Therefore, kernel logic is used for the transformation of data.  A non-linear kernel like the RBF kernel uses gamma value. It determines the influence of training points on the decision boundary. If the gamma value is low, all training points have an influence on the decision line, and the radius of similarity is high. If the gamma value is high, the radius of similarity is low and influence is limited only to the nearby points. A larger gamma value, therefore, results in overfitting. To learn more about gamma hyperparameter, refer to https://scikit-learn.org/stable/modules/svm.html#svm-kernels.

The following code snippet shows SVM with Grid search on breast -cancer dataset.

#import all necessary packages
import pandas as pd
import numpy as np
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.datasets import load_breast_cancerfrom sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
#load dataset  
cancer = load_breast_cancer()

#Converting to feature and target datasets
X = pd.DataFrame(cancer.data, columns=cancer.feature_names)
y=pd.DataFrame(cancer['target'])

#splitting into train and test data
X_train, X_test, y_train, y_test = train_test_split(
                        X, np.ravel(y),
                test_size = 0.3, random_state = 20


# defining parameter range
#parameter grid takes a dictionary containg list of values of each hyperparameter
param_grid = {'C': [0.1, 1, 10, 100, 1000], 
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel': ['rbf']} 


grid = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3)

# fitting the model for grid search
grid.fit(X_train, y_train)

# print best parameter after tuning
print(grid.best_params_)
#output
#{'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}

# print how our model looks after hyper-parameter tuning
print(grid.best_estimator_)
#output
#SVC(C=100, gamma=0.0001)

#testing the model
grid_predictions = grid.predict(X_test)
print(classification_report(y_test, grid_predictions))

 precision    recall  f1-score   support

           0       0.92      0.88      0.90        64
           1       0.93      0.95      0.94       107

    accuracy                           0.92       171
   macro avg       0.92      0.91      0.92       171
weighted avg       0.92      0.92      0.92       171

An accuracy of 95% is achieved with Grid Search which is a lot more higher than traditional SVM.