Visual Comparison Between Different Classification Methods in Shogun

Notebook by Youssef Emad El-Din (Github ID: youssef-emad)

This notebook demonstrates different classification methods in Shogun. The point is to compare and visualize the decision boundaries of different classifiers on two different datasets, where one is linear seperable, and one is not.

  1. <a href ="#section1">Data Generation and Visualization</a>
  2. <a href ="#section2">Support Vector Machine</a>
    1. <a href ="#section2a">Linear SVM</a>
    2. <a href ="#section2b">Gaussian Kernel</a>
    3. <a href ="#section2c">Sigmoid Kernel</a>
    4. <a href ="#section2d">Polynomial Kernel</a>
  3. <a href ="#section3">Naive Bayes</a>
  4. <a href ="#section4">Nearest Neighbors</a>
  5. <a href ="#section5">Linear Discriminant Analysis</a>
  6. <a href ="#section6">Quadratic Discriminat Analysis</a>
  7. <a href ="#section7">Gaussian Process</a>
    1. <a href ="#section7a">Logit Likelihood model</a>
    2. <a href ="#section7b">Probit Likelihood model</a>
  8. <a href ="#section8">Putting It All Together</a>
In [1]:
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from modshogun import *
In [2]:
#Needed lists for the final plot
classifiers_linear = []*10
classifiers_non_linear = []*10
classifiers_names = []*10
fadings = []*10

<a id = "section1">Data Generation and Visualization</a>

Transformation of features to Shogun format using RealFeatures and BinaryLables classes.

In [3]:
shogun_feats_linear = RealFeatures(CSVFile('../../../data/toy/classifier_binary_2d_linear_features_train.dat'))
shogun_labels_linear = BinaryLabels(CSVFile('../../../data/toy/classifier_binary_2d_linear_labels_train.dat'))

shogun_feats_non_linear = RealFeatures(CSVFile('../../../data/toy/classifier_binary_2d_nonlinear_features_train.dat'))
shogun_labels_non_linear = BinaryLabels(CSVFile('../../../data/toy/classifier_binary_2d_nonlinear_labels_train.dat'))

feats_linear = shogun_feats_linear.get_feature_matrix()
labels_linear = shogun_labels_linear.get_labels()

feats_non_linear = shogun_feats_non_linear.get_feature_matrix()
labels_non_linear = shogun_labels_non_linear.get_labels()

Data visualization methods.

In [4]:
def plot_binary_data(plot,X_train, y_train):
    """
    This function plots 2D binary data with different colors for different labels.
    """
    plot.xlabel(r"$x$")
    plot.ylabel(r"$y$")
    plot.plot(X_train[0, np.argwhere(y_train == 1)], X_train[1, np.argwhere(y_train == 1)], 'ro')
    plot.plot(X_train[0, np.argwhere(y_train == -1)], X_train[1, np.argwhere(y_train == -1)], 'bo')
In [5]:
def compute_plot_isolines(classifier,features,size=200,fading=True):
    """
    This function computes the classification of points on the grid
    to get the decision boundaries used in plotting
    """
    x1 = np.linspace(1.2*min(features[0]), 1.2*max(features[0]), size)
    x2 = np.linspace(1.2*min(features[1]), 1.2*max(features[1]), size)

    x, y = np.meshgrid(x1, x2)

    plot_features=RealFeatures(np.array((np.ravel(x), np.ravel(y))))
    
    if fading == True:
        plot_labels = classifier.apply(plot_features).get_values()
    else:
        plot_labels = classifier.apply(plot_features).get_labels()
    z = plot_labels.reshape((size, size))
    return x,y,z
In [6]:
def plot_model(plot,classifier,features,labels,fading=True):
    """
    This function plots an input classification model
    """
    x,y,z = compute_plot_isolines(classifier,features,fading=fading)
    plot.pcolor(x,y,z,cmap='RdBu_r')
    plot.contour(x, y, z, linewidths=1, colors='black')
    plot_binary_data(plot,features, labels)
    
In [7]:
plt.figure(figsize=(15,5))
plt.subplot(121)
plt.title("Linear Features")
plot_binary_data(plt,feats_linear, labels_linear)
plt.subplot(122)
plt.title("Non Linear Features")
plot_binary_data(plt,feats_non_linear, labels_non_linear)

Shogun provide Liblinear which is a library for large-scale linear learning focusing on SVM used for classification

In [8]:
plt.figure(figsize=(15,5))
c  = 0.5
epsilon =1e-3

svm_linear = LibLinear(c,shogun_feats_linear,shogun_labels_linear)
svm_linear.set_liblinear_solver_type(L2R_L2LOSS_SVC) 
svm_linear.set_epsilon(epsilon)
svm_linear.train()
classifiers_linear.append(svm_linear)
classifiers_names.append("SVM Linear")
fadings.append(True)

plt.subplot(121)
plt.title("Linear SVM - Linear Features")
plot_model(plt,svm_linear,feats_linear,labels_linear)

svm_non_linear = LibLinear(c,shogun_feats_non_linear,shogun_labels_non_linear)
svm_non_linear.set_liblinear_solver_type(L2R_L2LOSS_SVC) 
svm_non_linear.set_epsilon(epsilon)
svm_non_linear.train()
classifiers_non_linear.append(svm_non_linear)

plt.subplot(122)
plt.title("Linear SVM - Non Linear Features")
plot_model(plt,svm_non_linear,feats_non_linear,labels_non_linear)

SVM - Kernels

Shogun provides many options for using kernel functions. Kernels in Shogun are based on two classes which are CKernel and CKernelMachine base class.

In [9]:
gaussian_c=0.7

gaussian_kernel_linear=GaussianKernel(shogun_feats_linear, shogun_feats_linear, 100)
gaussian_svm_linear=LibSVM(gaussian_c, gaussian_kernel_linear, shogun_labels_linear)
gaussian_svm_linear.train()
classifiers_linear.append(gaussian_svm_linear)
fadings.append(True)

gaussian_kernel_non_linear=GaussianKernel(shogun_feats_non_linear, shogun_feats_non_linear, 100)
gaussian_svm_non_linear=LibSVM(gaussian_c, gaussian_kernel_non_linear, shogun_labels_non_linear)
gaussian_svm_non_linear.train()
classifiers_non_linear.append(gaussian_svm_non_linear)
classifiers_names.append("SVM Gaussian Kernel")

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.title("SVM Gaussian Kernel - Linear Features")
plot_model(plt,gaussian_svm_linear,feats_linear,labels_linear)

plt.subplot(122)
plt.title("SVM Gaussian Kernel - Non Linear Features")
plot_model(plt,gaussian_svm_non_linear,feats_non_linear,labels_non_linear)
In [10]:
sigmoid_c = 0.9

sigmoid_kernel_linear = SigmoidKernel(shogun_feats_linear,shogun_feats_linear,200,1,0.5)
sigmoid_svm_linear = LibSVM(sigmoid_c, sigmoid_kernel_linear, shogun_labels_linear)
sigmoid_svm_linear.train()
classifiers_linear.append(sigmoid_svm_linear)
classifiers_names.append("SVM Sigmoid Kernel")
fadings.append(True)

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.title("SVM Sigmoid Kernel - Linear Features")
plot_model(plt,sigmoid_svm_linear,feats_linear,labels_linear)

sigmoid_kernel_non_linear = SigmoidKernel(shogun_feats_non_linear,shogun_feats_non_linear,400,2.5,2)
sigmoid_svm_non_linear = LibSVM(sigmoid_c, sigmoid_kernel_non_linear, shogun_labels_non_linear)
sigmoid_svm_non_linear.train()
classifiers_non_linear.append(sigmoid_svm_non_linear)

plt.subplot(122)
plt.title("SVM Sigmoid Kernel - Non Linear Features")
plot_model(plt,sigmoid_svm_non_linear,feats_non_linear,labels_non_linear)