Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data…

• SU
  Shandong University
• NN
  National Natural Science Foundation of China

  Awards: 61972234, 62102234, 62272274, 62202271, 62072279
• CS
  China Scholarship Council
• NS
  Natural Science Foundation of Shandong Province

  Award: ZR2021QF129