An exploratory test for an excess of significant findings

The published clinical research literature may be distorted by the pursuit of statistically significant results. PURPOSE: We aimed to develop a test to explore biases stemming from the pursuit of nominal statistical significance.

The exploratory test evaluates whether there is a relative excess of formally significant findings in the published literature due to any reason (e.g., publication bias, selective analyses and outcome reporting, or fabricated data). The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies. The main application uses alpha = 0.05, but a range of alpha thresholds is also examined. Different values or prior distributions of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).