Linear Ridge Regression

A linear ridge regression model can be defined as \(y_i = \bf{w}^\top\bf{x_i}\) where \(y_i\) is the predicted value, \(\bf{x_i}\) is a feature vector and \(\bf{w}\) is the weight vector. We aim to find the linear function that best explains the data, i.e. minimizes the squared loss plus a \(L_2\) regularization term. One can show the solution can be written as:

\[{\bf w} = \left(\tau {\bf I}+ \sum_{i=1}^N{\bf x}_i{\bf x}_i^\top\right)^{-1}\left(\sum_{i=1}^N y_i{\bf x}_i\right)\]

where \(N\) is the number of training samples and \(\tau>0\) scales the regularization term.

A bias term, which is the squared empirical error, can also be calculated.

For the special case when \(\tau = 0\), a wrapper class CLeastSquaresRegression is available.

Example

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CRegressionLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = RegressionLabels(f_labels_train)
labels_test = RegressionLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = RegressionLabels(f_labels_train);
labels_test = RegressionLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);
features_train = Modshogun::RealFeatures.new f_feats_train
features_test = Modshogun::RealFeatures.new f_feats_test
labels_train = Modshogun::RegressionLabels.new f_labels_train
labels_test = Modshogun::RegressionLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- RegressionLabels(f_labels_train)
labels_test <- RegressionLabels(f_labels_test)
features_train = modshogun.RealFeatures(f_feats_train)
features_test = modshogun.RealFeatures(f_feats_test)
labels_train = modshogun.RegressionLabels(f_labels_train)
labels_test = modshogun.RegressionLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CRegressionLabels>(f_labels_train);
auto labels_test = some<CRegressionLabels>(f_labels_test);

We create an instance of CLinearRidgeRegression classifier, passing it \(\tau\), training data and labels.

tau = 0.001
lrr = LinearRidgeRegression(tau, features_train, labels_train)
tau = 0.001;
lrr = LinearRidgeRegression(tau, features_train, labels_train);
double tau = 0.001;
LinearRidgeRegression lrr = new LinearRidgeRegression(tau, features_train, labels_train);
tau = 0.001
lrr = Modshogun::LinearRidgeRegression.new tau, features_train, labels_train
tau <- 0.001
lrr <- LinearRidgeRegression(tau, features_train, labels_train)
tau = 0.001
lrr = modshogun.LinearRidgeRegression(tau, features_train, labels_train)
double tau = 0.001;
LinearRidgeRegression lrr = new LinearRidgeRegression(tau, features_train, labels_train);
auto tau = 0.001;
auto lrr = some<CLinearRidgeRegression>(tau, features_train, labels_train);

Then we train the regression model and apply it to test data to get the predicted CRegressionLabels and bias.

lrr.train()
labels_predict = lrr.apply_regression(features_test)
bias = lrr.get_bias()
lrr.train();
labels_predict = lrr.apply_regression(features_test);
bias = lrr.get_bias();
lrr.train();
RegressionLabels labels_predict = lrr.apply_regression(features_test);
double bias = lrr.get_bias();
lrr.train 
labels_predict = lrr.apply_regression features_test
bias = lrr.get_bias 
lrr$train()
labels_predict <- lrr$apply_regression(features_test)
bias <- lrr$get_bias()
lrr:train()
labels_predict = lrr:apply_regression(features_test)
bias = lrr:get_bias()
lrr.train();
RegressionLabels labels_predict = lrr.apply_regression(features_test);
double bias = lrr.get_bias();
lrr->train();
auto labels_predict = lrr->apply_regression(features_test);
auto bias = lrr->get_bias();

Optionally the bias can be disabled to avoid redundant computation.

lrr.set_compute_bias(False)
lrr.set_compute_bias(false);
lrr.set_compute_bias(false);
lrr.set_compute_bias 0
lrr$set_compute_bias(FALSE)
lrr:set_compute_bias(False)
lrr.set_compute_bias(false);
lrr->set_compute_bias(false);

Imagine, we know the bias term. We can set it as

lrr.set_bias(bias)
lrr.set_bias(bias);
lrr.set_bias(bias);
lrr.set_bias bias
lrr$set_bias(bias)
lrr:set_bias(bias)
lrr.set_bias(bias);
lrr->set_bias(bias);

After training, we can extract \({\bf w}\).

weights = lrr.get_w()
weights = lrr.get_w();
DoubleMatrix weights = lrr.get_w();
weights = lrr.get_w 
weights <- lrr$get_w()
weights = lrr:get_w()
double[] weights = lrr.get_w();
auto weights = lrr->get_w();

Finally, we can evaluate the CMeanSquaredError.

eval = MeanSquaredError()
mse = eval.evaluate(labels_predict, labels_test)
eval = MeanSquaredError();
mse = eval.evaluate(labels_predict, labels_test);
MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);
eval = Modshogun::MeanSquaredError.new 
mse = eval.evaluate labels_predict, labels_test
eval <- MeanSquaredError()
mse <- eval$evaluate(labels_predict, labels_test)
eval = modshogun.MeanSquaredError()
mse = eval:evaluate(labels_predict, labels_test)
MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);
auto eval = some<CMeanSquaredError>();
auto mse = eval->evaluate(labels_predict, labels_test);