Evaluating Machine Learning Models and Their Diagnostic Value

Abstract This chapter describes model validation, a crucial part of machine learning whether it is to select the best model or to assess performance of a given model. We start by detailing the main performance metrics for different tasks (classification, regression), and how they may be interpreted, including in the face of class imbalance, varying prevalence, or asymmetric cost–benefit trade-offs. We then explain how to estimate these metrics in an unbiased manner using training, validation, and test sets. We describe cross-validation procedures—to use a larger part of the data for both training and testing—and the dangers of data leakage—optimism bias due to training data contaminating the test set. Finally,…

• AN
  Agence Nationale de la Recherche

  Awards: ANR-20-CHIA-0026, ANR-10-IAIHU-06, 10-IAIHU-06, ANR-19-P3IA-0001, 19-P3IA-0001, ANR-10, ANR-19