base class for cross-validation evaluation. Given a learning machine, a splitting strategy, an evaluation criterium, features and correspnding labels, this provides an interface for cross-validation. Results may be retrieved using the evaluate method. A number of repetitions may be specified for obtaining more accurate results. The arithmetic mean of different runs is returned along with confidence intervals, if a p-value is specified. Default number of runs is one, confidence interval combutation is disabled.
This class calculates an evaluation criterium of every fold and then calculates the arithmetic mean of all folds. This is for example suitable for the AUC or for Accuracy. However, for example F1-measure may not be merged this way (result will be biased). To solve this, different sub-classes may average results of each cross validation fold differently by overwriting the evaluate_one_run method.
See [Forman, G. and Scholz, M. (2009). Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. Technical report, HP Laboratories.] for details on this subject.
Definition at line 77 of file CrossValidation.h.
List of all members.
Constructor & Destructor Documentation
|machine ||learning machine to use |
|features ||features to use for cross-validation |
|labels ||labels that correspond to the features |
|splitting_strategy ||splitting strategy to use |
|evaluation_criterium ||evaluation criterium to use |
Definition at line 34 of file CrossValidation.cpp.
Member Function Documentation
method for evaluation. Performs cross-validation. Is repeated m_num_runs. If this number is larger than one, a confidence interval is calculated if m_conf_int_alpha is (0<p<1). By default m_num_runs=1 and m_conf_int_alpha=0
- result of evaluation
Definition at line 82 of file CrossValidation.cpp.
Evaluates one single cross-validation run. Current implementation evaluates each fold separately and then calculates arithmetic mean. Suitable for accuracy and AUC for example. NOT for F1-measure. Has to be overridden by sub-classes if results have to be merged differently
- evaluation result of one cross-validation run
Definition at line 131 of file CrossValidation.cpp.
- in which direction is the best evaluation value?
Definition at line 98 of file CrossValidation.h.
|virtual const char* get_name
The documentation for this class was generated from the following files: