A prevalent problem when utilizing machine learning algorithms is overfitting, or when an algorithm “memorizes” the training data but does a poor job extrapolating to out of sample cases. A common method for dealing with the overfitting problem is to hold back some subset of data from the original training algorithm and then measure the fit algorithm’s performance on this holdout set. This is commonly known as cross validation. A model is trained on one subset of data and then validated on another set of data.
There are several strategies for holding out data. FlinkML has convenience methods for
The simplest method of splitting is the trainTestSplit
. This split takes a DataSet and a parameter fraction. The fraction indicates the portion of the DataSet that should be allocated to the training set. This split also takes two additional optional parameters, precise and seed.
By default, the Split is done by randomly deciding whether or not an observation is assigned to the training DataSet with probability = fraction. When precise is true
however, additional steps are taken to ensure the training set is as close as possible to the length of the DataSet $\cdot$ fraction.
The method returns a new TrainTestDataSet
object which has a .training
attribute containing the training DataSet and a .testing
attribute containing the testing DataSet.
In some cases, algorithms have been known to ‘learn’ the testing set. To combat this issue, a traintesthold out strategy introduces a secondary holdout set, aptly called the holdout set.
Traditionally, training and testing would be done to train an algorithms as normal and then a final test of the algorithm on the holdout set would be done. Ideally, prediction errors/model scores in the holdout set would not be significantly different than those observed in the testing set.
In a traintestholdout strategy we sacrifice the sample size of the initial fitting algorithm for increased confidence that our model is not overfit.
When using trainTestHoldout
splitter, the fraction Double
is replaced by a fraction array of length three. The first element corresponds to the portion to be used for training, second for testing, and third for holdout. The weights of this array are relative, e.g. an array Array(3.0, 2.0, 1.0)
would results in approximately 50% of the observations being in the training set, 33% of the observations in the testing set, and 17% of the observations in holdout set.
In a kfold strategy, the DataSet is split into k equal subsets. Then for each of the k subsets, a TrainTestDataSet
is created where the subset is the .training
DataSet, and the remaining subsets are the .testing
set.
For each training set, an algorithm is trained and then is evaluated based on the predictions based on the associated testing set. When an algorithm that has consistent grades (e.g. prediction errors) across held out datasets we can have some confidence that our approach (e.g. choice of algorithm / algorithm parameters / number of iterations) is robust against overfitting.
The multirandom strategy can be thought of as a more general form of the traintestholdout strategy. In fact, .trainTestHoldoutSplit
is a simple wrapper for multiRandomSplit
which also packages the datasets into a trainTestHoldoutDataSet
object.
The first major difference, is that multiRandomSplit
takes an array of fractions of any length. E.g. one can create multiple holdout sets. Alternatively, one could think of kFoldSplit
as a wrapper for multiRandomSplit
(which it is), the difference being kFoldSplit
creates subsets of approximately equal size, where multiRandomSplit
will create subsets of any size.
The second major difference is that multiRandomSplit
returns an array of DataSets, equal in size and proportion to the fraction array that it was passed as an argument.
The various Splitter
methods share many parameters.
Parameter  Type  Description  Used by Method 

input 
DataSet[Any] 
DataSet to be split. 
randomSplit multiRandomSplit kFoldSplit trainTestSplit trainTestHoldoutSplit

seed 
Long 
Used for seeding the random number generator which sorts DataSets into other DataSets. 
randomSplit multiRandomSplit kFoldSplit trainTestSplit trainTestHoldoutSplit

precise 
Boolean 
When true, make additional effort to make DataSets as close to the prescribed proportions as possible. 
randomSplit trainTestSplit

fraction 
Double 
The portion of the `input` to assign to the first or .training DataSet. Must be in the range (0,1) 
randomSplit trainTestSplit

fracArray 
Array[Double] 
An array that prescribes the proportions of the output datasets (proportions need not sum to 1 or be within the range (0,1)) 
multiRandomSplit trainTestHoldoutSplit

kFolds 
Int 
The number of subsets to break the input DataSet into. 
kFoldSplit 