Grid Search for Hyperparameter tuning in SVM using scikit-learn
SVM Introduction
In this article, I will show you how to improve your SVM(Support Vector Machine) model’s accuracy or other metrics, using Grid Search using scikit-learn in Python. SVM chooses a hyperplane(best region or boundary) to segregate the objects into distinct classes. The points of each class very close to the hyperplane are called support vectors. The distance between these vectors and hyperplane is called a margin. Working of an SVM when there are only 2 classes in a 2d plane is shown below. This is an example of a linear boundary SVM.
Linear SVM Classifier.
The SVM, as you know is a supervised machine learning algorithm that chooses the decision boundary by taking into consideration the following:
a)Increase the distance of the decision boundary from support vectors, also known as margin.
b)Minimise the number of misclassified items.
Hyperparameters in SVM
A machine learning algorithm requires certain hyperparameters that must be tuned before training. An optimal subset of these hyperparameters must be selected, which is called hyperparameter optimization. Grid-Search is a sci-kit learn package that provides for hyperparameter tuning. A grid search space is generated by taking the initial set of values given to each hyperparameter. Each cell in the grid is searched for the optimal solution.
There are two hyperparameters to be tuned on an SVM model: C and gamma.
C value: C value adds a penalty each time an item is misclassified. So, a low C value has more misclassified items. On the other hand, a large C value has a plane with a very little margin which leads to overfitting.
γ value: It is not always possible to classify data using a linear kernel when data points are complexly arranged. Therefore, kernel logic is used for the transformation of data. A non-linear kernel like the RBF kernel uses gamma value. It determines the influence of training points on the decision boundary. If the gamma value is low, all training points have an influence on the decision line, and the radius of similarity is high. If the gamma value is high, the radius of similarity is low and influence is limited only to the nearby points. A larger gamma value, therefore, results in overfitting. To learn more about gamma hyperparameter, refer to https://scikit-learn.org/stable/modules/svm.html#svm-kernels.
The following code snippet shows SVM with Grid search on breast -cancer dataset.
#import all necessary packages import pandas as pd import numpy as np from sklearn.metrics import classification_report, confusion_matrix from sklearn.datasets import load_breast_cancerfrom sklearn.model_selection import train_test_split from sklearn.svm import SVC from sklearn.model_selection import GridSearchCV #load dataset cancer = load_breast_cancer() #Converting to feature and target datasets X = pd.DataFrame(cancer.data, columns=cancer.feature_names) y=pd.DataFrame(cancer['target']) #splitting into train and test data X_train, X_test, y_train, y_test = train_test_split( X, np.ravel(y), test_size = 0.3, random_state = 20 # defining parameter range #parameter grid takes a dictionary containg list of values of each hyperparameter param_grid = {'C': [0.1, 1, 10, 100, 1000], 'gamma': [1, 0.1, 0.01, 0.001, 0.0001], 'kernel': ['rbf']} grid = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3) # fitting the model for grid search grid.fit(X_train, y_train) # print best parameter after tuning print(grid.best_params_) #output #{'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'} # print how our model looks after hyper-parameter tuning print(grid.best_estimator_) #output #SVC(C=100, gamma=0.0001) #testing the model grid_predictions = grid.predict(X_test) print(classification_report(y_test, grid_predictions))
precision recall f1-score support 0 0.92 0.88 0.90 64 1 0.93 0.95 0.94 107 accuracy 0.92 171 macro avg 0.92 0.91 0.92 171 weighted avg 0.92 0.92 0.92 171
An accuracy of 95% is achieved with Grid Search which is a lot more higher than traditional SVM.
Leave a Reply