Statistics, Handle with Care: Detecting Multiple Model Components with the Likelihood Ratio Test

The likelihood ratio test (LRT) and the related $F$ test, do not (even asymptotically) adhere to their nominal $\\chi^2$ and $F$ distributions in many statistical tests common in astrophysics, thereby casting many marginal line or source detections and non-detections into doubt. Although there are many legitimate uses of these statistics, in some important cases it can be impossible to compute the correct false positive rate. For example, it has become common practice to use the LRT or the $F$ test for detecting a line in a spectral model or a source above background despite the lack of certain required regularity conditions. In these and other settings that involve testing a hypothesis that is on the boundary…