Cross-validation is a method for estimating the performance of a machine learning model over new, unseen samples of data.

At the Discover toolkit, after a predictive or evolutive machine learning model is trained, the model's results page will display cross-validation (CV) results in the summary card:

The score inside the red rectangle can be seen as an estimation of the metric R2 over a novel dataset. This cross-validated metric is very helpful as all the data provided in model's creation was used for training, and the training scores will have better results than scores calculated with new samples.

A detailed view of CV results is displayed within the Performance Statistics card. This card shows all scoring metrics calculated with the CV approach, as seen below:

The above view show the statistics of the different metrics that were calculated using the CV approach. A more detailed explanation on how those statistics are calculated is discussed within the following title.

Cross-validation (CV) [1] is any of various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (called the validation dataset or testing set). The goal of cross-validation is to test the model's ability to predict new data that was not used in training, in order to flag problems like overfitting or selection bias and to give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance from a real problem).

One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to give an estimate of the model's predictive performance.

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used, but in general k remains an unfixed parameter.

In repeated k-fold CV the data is randomly split into k partitions r times. The performance of the model can thereby be averaged over several runs. Repetition is useful to reduce variance in score estimation, and especially useful when datasets are small (<= 200 samples).

By default DISCOVER uses k = 5 and r = 5.

In DISCOVER, CV metrics are displayed in terms of their statistics over the many CV partitions. These statistics are:

FAQ:

Not right now, but you can ask the TeselaGen’s Data Science team to do it for you. In future releases we are planning to enable configurable CV parameters to our users.

[1] wikipedia: https://en.wikipedia.org/wiki/Cross-validation_(statistics)

More DISCOVER help articles: