How to get statistically significant effects in any ERP experiment (and why you shouldn't)

ERP experiments generate massive datasets, often containing thousands of values for each participant, even after averaging. The richness of these datasets can be very useful in testing sophisticated hypotheses, but this richness also creates many opportunities to obtain effects that are statistically significant but do not reflect true differences among groups or conditions (bogus effects). The purpose of this paper is to demonstrate how common and seemingly innocuous methods for quantifying and analyzing ERP effects can lead to very high rates of significant but bogus effects, with the likelihood of obtaining at least one such bogus effect exceeding 50% in many experiments. We focus on two specific problems:…

• NI
  National Institute of Mental Health

  Awards: R01MH076226, R25MH080794
• NE
  National Eye Institute

  Award: F32EY024834