Using cognitive psychology to understand GPT-3

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails…

• DF
  Deutsche Forschungsgemeinschaft

  Awards: EXC2064, EXC2064/1-390727645, 390727645
• VF
  Volkswagen Foundation

  Award: VW98569
• M
  Max-Planck-Gesellschaft