When Benchmark Results Do Not License Strong Claims: Same-Channel Non-Repairability in AI Evaluation

This article examines a specific problem in AI evaluation: cases in which strong capability claims are drawn from results that do not, by themselves, justify those claims. Its central argument is that some benchmark results, behavioural outputs, and evaluative signals fail to support stronger conclusions not merely because the evidence is weak, but because the result-channel does not preserve the distinctions required for those conclusions. The paper introduces a minimal criterion for identifying such cases, described here as same-channel non-repairability, and develops it through a case study centred on recent debates about reasoning-model evaluation, including The Illusion of Thinking, subsequent rebuttals,…