Is K-fold cross validation the best model selection method for machine learning?

As a technique that can compactly represent complex patterns, machine learning has significant potential for predictive inference in multi-source and heterogeneous information fusion scenarios. K-fold cross-validation (CV) is the most common approach for ascertaining the likelihood that a machine learning outcome is generated by chance and frequently outperforms conventional hypothesis testing. This improvement arises from measures directly obtained from machine learning classifications, such as accuracy, that do not have a parametric description. To approach a frequentist analysis within fusion-oriented machine learning pipelines, a permutation test or simple statistics from data partitions (i.e., folds) can…

• MD
  Ministerio de Ciencia, Innovación y Universidades
• JD
  Junta de Andalucía

  Awards: UGR20, UGR18
• PA
  Pan African Materials Institute
• ER
  European Regional Development Fund

  Award: MICIU/AEI/10
• AE
  Agencia Estatal de Investigación

  Awards: AEI/10.13039/501100011033, 501100011033, 10.13039/501100011033, MCIN/ AEI/10.13039/501100011033, 13039, 10.13039, AEI/10, 13039/501100011033, AEI/10.