base class for cross-validation evaluation. Given a learning machine, a splitting strategy, an evaluation criterium, features and correspnding labels, this provides an interface for cross-validation. Results may be retrieved using the evaluate method. A number of repetitions may be specified for obtaining more accurate results. The arithmetic mean of different runs is returned along with confidence intervals, if a p-value is specified. Default number of runs is one, confidence interval combutation is disabled.
This class calculates an evaluation criterium of every fold and then calculates the arithmetic mean of all folds. This is for example suitable for the AUC or for Accuracy. However, for example F1-measure may not be merged this way (result will be biased). To solve this, different sub-classes may average results of each cross validation fold differently by overwriting the evaluate_one_run method.
See [Forman, G. and Scholz, M. (2009). Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. Technical report, HP Laboratories.] for details on this subject.
Definition at line 77 of file CrossValidation.h.
Public Member Functions | |
CCrossValidation () | |
CCrossValidation (CMachine *machine, CFeatures *features, CLabels *labels, CSplittingStrategy *splitting_strategy, CEvaluation *evaluation_criterium) | |
virtual | ~CCrossValidation () |
EEvaluationDirection | get_evaluation_direction () |
CrossValidationResult | evaluate () |
CMachine * | get_machine () const |
void | set_num_runs (int32_t num_runs) |
void | set_conf_int_alpha (float64_t m_conf_int_alpha) |
virtual const char * | get_name () const |
Protected Member Functions | |
virtual float64_t | evaluate_one_run () |
CCrossValidation | ( | ) |
constructor
Definition at line 20 of file CrossValidation.cpp.
CCrossValidation | ( | CMachine * | machine, | |
CFeatures * | features, | |||
CLabels * | labels, | |||
CSplittingStrategy * | splitting_strategy, | |||
CEvaluation * | evaluation_criterium | |||
) |
constructor
machine | learning machine to use | |
features | features to use for cross-validation | |
labels | labels that correspond to the features | |
splitting_strategy | splitting strategy to use | |
evaluation_criterium | evaluation criterium to use |
Definition at line 34 of file CrossValidation.cpp.
~CCrossValidation | ( | ) | [virtual] |
destructor
Definition at line 25 of file CrossValidation.cpp.
CrossValidationResult evaluate | ( | ) |
method for evaluation. Performs cross-validation. Is repeated m_num_runs. If this number is larger than one, a confidence interval is calculated if m_conf_int_alpha is (0<p<1). By default m_num_runs=1 and m_conf_int_alpha=0
Definition at line 82 of file CrossValidation.cpp.
float64_t evaluate_one_run | ( | ) | [protected, virtual] |
Evaluates one single cross-validation run. Current implementation evaluates each fold separately and then calculates arithmetic mean. Suitable for accuracy and AUC for example. NOT for F1-measure. Has to be overridden by sub-classes if results have to be merged differently
Definition at line 131 of file CrossValidation.cpp.
EEvaluationDirection get_evaluation_direction | ( | ) |
Definition at line 98 of file CrossValidation.h.
CMachine * get_machine | ( | ) | const |
Definition at line 76 of file CrossValidation.cpp.
virtual const char* get_name | ( | void | ) | const [virtual] |
Implements CSGObject.
Definition at line 122 of file CrossValidation.h.
void set_conf_int_alpha | ( | float64_t | m_conf_int_alpha | ) |
setter for the number of runs to use for evaluation
Definition at line 112 of file CrossValidation.cpp.
void set_num_runs | ( | int32_t | num_runs | ) |
setter for the number of runs to use for evaluation
Definition at line 123 of file CrossValidation.cpp.