Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT

Abstract We design a battery of semantic illusions and cognitive reflection tests, aimed to elicit intuitive yet erroneous responses. We administer these tasks, traditionally used to study reasoning and decision-making in humans, to OpenAI’s generative pre-trained transformer model family. The results show that as the models expand in size and linguistic proficiency they increasingly display human-like intuitive system 1 thinking and associated cognitive errors. This pattern shifts notably with the introduction of ChatGPT models, which tend to respond correctly, avoiding the traps embedded in the tasks. Both ChatGPT-3.5 and 4 utilize the input–output context window to engage in chain-of-thought reasoning,…

• MF
  Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg

  Award: Az. 33-7533-9-19/54/5
• US
  Universität Stuttgart