Reliable Accuracy Estimates from k -Fold Cross Validation

It is popular to evaluate the performance of classification algorithms by k-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k-fold cross validation. Most of them did not consider the correlation among the replications of k-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k-fold cross validation are first analyzed for k-nearest neighbors with k =…

• MO
  Ministry of Science and Technology, Taiwan

  Award: 106-2410-H-006-020