The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences

Individual differences in cognitive paradigms are increasingly employed to relate cognition to brain structure, chemistry, and function. However, such efforts are often unfruitful, even with the most well established tasks. Here we offer an explanation for failures in the application of robust cognitive paradigms to the study of individual differences. Experimental effects become well established - and thus those tasks become popular - when between-subject variability is low. However, low between-subject variability causes low reliability for individual differences, destroying replicable correlations with other factors and potentially undermining published conclusions drawn from correlational relationships.…

• W
  Wellcome

  Award: 104943/Z/14/Z
• EA
  Economic and Social Research Council

  Awards: ES/K002325/1, ES/K002325/1