Linear Discriminant Analysis

LDA learns a linear classifier via finding a projection matrix that maximally discriminates the provided classes. The learned linear classification rule is optimal under the assumption that both classes a gaussian distributed with equal co-variance. To find a linear separation \({\bf w}\) in training, the in-between class variance is maximized and the within class variance is minimized. The projection matrix is computed by maximizing the following objective:

\[J({\bf w})=\frac{{\bf w^T} S_B {\bf w}}{{\bf w^T} S_W {\bf w}}\]

where \({\bf S_B}\) is between class scatter matrix and \({\bf S_W}\) is within class scatter matrix. The above derivation of LDA requires the invertibility of the within class matrix. This condition however, is violated when there are fewer data-points than dimensions. In this case SVD is used to compute projection matrix using an orthonormal basis \({\bf Q}\)

\[{\bf W} := {\bf Q} {\bf{W^\prime}}\]

See Chapter 16 in [Bar12] for a detailed introduction.

Example

We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CBinaryLabels from files with training and test data.

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = BinaryLabels(f_labels_train)
labels_test = BinaryLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = BinaryLabels(f_labels_train);
labels_test = BinaryLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::BinaryLabels.new f_labels_train
labels_test = Modshogun::BinaryLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- BinaryLabels(f_labels_train)
labels_test <- BinaryLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.BinaryLabels(f_labels_train)
labels_test = modshogun.BinaryLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CBinaryLabels>(f_labels_train);
auto labels_test = some<CBinaryLabels>(f_labels_test);

We create an instance of the CLDA classifier and set features and labels. By default, Shogun automatically chooses the decomposition method based on \({N<=D}\) or \({N>D}\).

lda = LDA()
lda.set_features(features_train)
lda.set_labels(labels_train)
lda = LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
lda = Modshogun::LDA.new 
lda.set_features features_train
lda.set_labels labels_train
lda <- LDA()
lda$set_features(features_train)
lda$set_labels(labels_train)
lda = modshogun.LDA()
lda:set_features(features_train)
lda:set_labels(labels_train)
LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
auto lda = some<CLDA>();
lda->set_features(features_train);
lda->set_labels(labels_train);

Then we train and apply it to test data, which here gives CBinaryLabels.

lda.train()
labels_predict = lda.apply_binary(features_test)
lda.train();
labels_predict = lda.apply_binary(features_test);
lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);
lda.train 
labels_predict = lda.apply_binary features_test
lda$train()
labels_predict <- lda$apply_binary(features_test)
lda:train()
labels_predict = lda:apply_binary(features_test)
lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);
lda->train();
auto labels_predict = lda->apply_binary(features_test);

We can extract weights \({\bf w}\).

w = lda.get_w()
w = lda.get_w();
DoubleMatrix w = lda.get_w();
w = lda.get_w 
w <- lda$get_w()
w = lda:get_w()
double[] w = lda.get_w();
auto w = lda->get_w();

We can evaluate test performance via e.g. CAccuracyMeasure.

eval = AccuracyMeasure()
accuracy = eval.evaluate(labels_predict, labels_test)
eval = AccuracyMeasure();
accuracy = eval.evaluate(labels_predict, labels_test);
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
eval = Modshogun::AccuracyMeasure.new 
accuracy = eval.evaluate labels_predict, labels_test
eval <- AccuracyMeasure()
accuracy <- eval$evaluate(labels_predict, labels_test)
eval = modshogun.AccuracyMeasure()
accuracy = eval:evaluate(labels_predict, labels_test)
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
auto eval = some<CAccuracyMeasure>();
auto accuracy = eval->evaluate(labels_predict, labels_test);

References

Wikipedia: Linear_discriminant_analysis

[Bar12]David Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.