Gaussian Process Classifier

Application of Gaussian processes in binary and multi-class classification. See Gaussian process regression cookbook and [RW05] for more information on Gaussian processes.

Example

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CMulticlassLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = MulticlassLabels(f_labels_train)
labels_test = MulticlassLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = MulticlassLabels(f_labels_train);
labels_test = MulticlassLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::MulticlassLabels.new f_labels_train
labels_test = Modshogun::MulticlassLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- MulticlassLabels(f_labels_train)
labels_test <- MulticlassLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.MulticlassLabels(f_labels_train)
labels_test = modshogun.MulticlassLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CMulticlassLabels>(f_labels_train);
auto labels_test = some<CMulticlassLabels>(f_labels_test);

To fit the input (training) data \(\mathbf{X}\), we have to choose appropriate CMeanFunction and CKernel. Here we use a basic CConstMean and a CGaussianKernel with chosen width parameter.

kernel = GaussianKernel(2.0)
mean_function = ConstMean()
kernel = GaussianKernel(2.0);
mean_function = ConstMean();
GaussianKernel kernel = new GaussianKernel(2.0);
ConstMean mean_function = new ConstMean();
kernel = Modshogun::GaussianKernel.new 2.0
mean_function = Modshogun::ConstMean.new 
kernel <- GaussianKernel(2.0)
mean_function <- ConstMean()
kernel = modshogun.GaussianKernel(2.0)
mean_function = modshogun.ConstMean()
GaussianKernel kernel = new GaussianKernel(2.0);
ConstMean mean_function = new ConstMean();
auto kernel = some<CGaussianKernel>(2.0);
auto mean_function = some<CConstMean>();

We need to specify the inference method to find the posterior distribution of the function values \(\mathbf{f}\). Here we choose to perform Laplace approximation inference method with an instance of CMultiLaplaceInferenceMethod (See Chapter 18.2 in [Bar12] for a detailed introduction) and pass it the chosen kernel, the training features, the mean function, the labels and an instance of CSoftMaxLikelihood, to specify the distribution of the targets/labels as above. Finally we create an instance of the CGaussianProcessClassification classifier.

gauss_likelihood = SoftMaxLikelihood()
inference_method = MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier = GaussianProcessClassification(inference_method)
gauss_likelihood = SoftMaxLikelihood();
inference_method = MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
gp_classifier = GaussianProcessClassification(inference_method);
SoftMaxLikelihood gauss_likelihood = new SoftMaxLikelihood();
MultiLaplaceInferenceMethod inference_method = new MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
GaussianProcessClassification gp_classifier = new GaussianProcessClassification(inference_method);
gauss_likelihood = Modshogun::SoftMaxLikelihood.new 
inference_method = Modshogun::MultiLaplaceInferenceMethod.new kernel, features_train, mean_function, labels_train, gauss_likelihood
gp_classifier = Modshogun::GaussianProcessClassification.new inference_method
gauss_likelihood <- SoftMaxLikelihood()
inference_method <- MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier <- GaussianProcessClassification(inference_method)
gauss_likelihood = modshogun.SoftMaxLikelihood()
inference_method = modshogun.MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier = modshogun.GaussianProcessClassification(inference_method)
SoftMaxLikelihood gauss_likelihood = new SoftMaxLikelihood();
MultiLaplaceInferenceMethod inference_method = new MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
GaussianProcessClassification gp_classifier = new GaussianProcessClassification(inference_method);
auto gauss_likelihood = some<CSoftMaxLikelihood>();
auto inference_method = some<CMultiLaplaceInferenceMethod>(kernel, features_train, mean_function, labels_train, gauss_likelihood);
auto gp_classifier = some<CGaussianProcessClassification>(inference_method);

Then we can train the model and evaluate the predictive distribution. We get predicted CMulticlassLabels.

gp_classifier.train()
labels_predict = gp_classifier.apply_multiclass(features_test)
gp_classifier.train();
labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier.train();
MulticlassLabels labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier.train 
labels_predict = gp_classifier.apply_multiclass features_test
gp_classifier$train()
labels_predict <- gp_classifier$apply_multiclass(features_test)
gp_classifier:train()
labels_predict = gp_classifier:apply_multiclass(features_test)
gp_classifier.train();
MulticlassLabels labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier->train();
auto labels_predict = gp_classifier->apply_multiclass(features_test);

We can extract the probabilities:

prob = gp_classifier.get_probabilities(features_train)
prob = gp_classifier.get_probabilities(features_train);
DoubleMatrix prob = gp_classifier.get_probabilities(features_train);
prob = gp_classifier.get_probabilities features_train
prob <- gp_classifier$get_probabilities(features_train)
prob = gp_classifier:get_probabilities(features_train)
double[] prob = gp_classifier.get_probabilities(features_train);
auto prob = gp_classifier->get_probabilities(features_train);

We can evaluate test performance via e.g. CMulticlassAccuracy.

evals = MulticlassAccuracy()
accuracy = evals.evaluate(labels_predict, labels_test)
evals = MulticlassAccuracy();
accuracy = evals.evaluate(labels_predict, labels_test);
MulticlassAccuracy evals = new MulticlassAccuracy();
double accuracy = evals.evaluate(labels_predict, labels_test);
evals = Modshogun::MulticlassAccuracy.new 
accuracy = evals.evaluate labels_predict, labels_test
evals <- MulticlassAccuracy()
accuracy <- evals$evaluate(labels_predict, labels_test)
evals = modshogun.MulticlassAccuracy()
accuracy = evals:evaluate(labels_predict, labels_test)
MulticlassAccuracy evals = new MulticlassAccuracy();
double accuracy = evals.evaluate(labels_predict, labels_test);
auto evals = some<CMulticlassAccuracy>();
auto accuracy = evals->evaluate(labels_predict, labels_test);

References

Wikipedia: Gaussian_process

[Bar12]David Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
[RW05]C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.