Multi-class Logistic Regression

Multinomial logistic regression assigns the sample \(\mathbf{x}_i\) to class \(c\) based on the probability for sample \(\mathbf{x}_i\) to be in class \(c\):

\[P(Y_i = c | \mathbf{x}_i) = \frac{\exp(\mathbf{\theta}^\top_c\mathbf{x}_i)}{1+ \sum_{k=1}^{K}\exp(\mathbf{\theta}^\top_k\mathbf{x}_i)}\]

in which \(K\) is the number of classes.

The loss function that needs to be minimized is:

\[{\min_{\mathbf{\theta}}}\sum_{k=1}^{K}\sum_{i=1}^{m}w_{ik}\log(1+\exp(-y_{ik}(\mathbf{x}_k^\top\mathbf{a}_{ik} + c_k))) + \lambda\left \| \mathbf{x} \right \|_{l_1/l_q}\]

where \(\mathbf{a}_{ik}\) denotes the \(i\)-th sample for the \(k\)-th class, \(w_{ik}\) is the weight for \(\mathbf{a}_{ik}^\top\), \(y_{ik}\) is the response of \(\mathbf{a}_{ik}\), and \(c_k\) is the intercept (scalar) for the \(k\)-th class. \(\lambda\) is the \(l_1/l_q\)-norm regularization parameter.

Example

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CMulticlassLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = MulticlassLabels(f_labels_train)
labels_test = MulticlassLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = MulticlassLabels(f_labels_train);
labels_test = MulticlassLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::MulticlassLabels.new f_labels_train
labels_test = Modshogun::MulticlassLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- MulticlassLabels(f_labels_train)
labels_test <- MulticlassLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.MulticlassLabels(f_labels_train)
labels_test = modshogun.MulticlassLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CMulticlassLabels>(f_labels_train);
auto labels_test = some<CMulticlassLabels>(f_labels_test);

We create an instance of the CMulticlassLogisticRegression classifier by passing it the dataset, lables, and specifying the regularization constant \(\lambda\) for each machine

classifier = MulticlassLogisticRegression(1, features_train, labels_train)
classifier = MulticlassLogisticRegression(1, features_train, labels_train);
MulticlassLogisticRegression classifier = new MulticlassLogisticRegression(1, features_train, labels_train);
classifier = Modshogun::MulticlassLogisticRegression.new 1, features_train, labels_train
classifier <- MulticlassLogisticRegression(1, features_train, labels_train)
classifier = modshogun.MulticlassLogisticRegression(1, features_train, labels_train)
MulticlassLogisticRegression classifier = new MulticlassLogisticRegression(1, features_train, labels_train);
auto classifier = some<CMulticlassLogisticRegression>(1, features_train, labels_train);

Then we train and apply it to test data, which here gives CMulticlassLabels.

classifier.train()
labels_predict = classifier.apply_multiclass(features_test)
classifier.train();
labels_predict = classifier.apply_multiclass(features_test);
classifier.train();
MulticlassLabels labels_predict = classifier.apply_multiclass(features_test);
classifier.train 
labels_predict = classifier.apply_multiclass features_test
classifier$train()
labels_predict <- classifier$apply_multiclass(features_test)
classifier:train()
labels_predict = classifier:apply_multiclass(features_test)
classifier.train();
MulticlassLabels labels_predict = classifier.apply_multiclass(features_test);
classifier->train();
auto labels_predict = classifier->apply_multiclass(features_test);

We can evaluate test performance via e.g. CMulticlassAccuracy.

eval = MulticlassAccuracy()
accuracy = eval.evaluate(labels_predict, labels_test)
eval = MulticlassAccuracy();
accuracy = eval.evaluate(labels_predict, labels_test);
MulticlassAccuracy eval = new MulticlassAccuracy();
double accuracy = eval.evaluate(labels_predict, labels_test);
eval = Modshogun::MulticlassAccuracy.new 
accuracy = eval.evaluate labels_predict, labels_test
eval <- MulticlassAccuracy()
accuracy <- eval$evaluate(labels_predict, labels_test)
eval = modshogun.MulticlassAccuracy()
accuracy = eval:evaluate(labels_predict, labels_test)
MulticlassAccuracy eval = new MulticlassAccuracy();
double accuracy = eval.evaluate(labels_predict, labels_test);
auto eval = some<CMulticlassAccuracy>();
auto accuracy = eval->evaluate(labels_predict, labels_test);