Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering

Knowledge-based visual question answering (VQA) requires external knowledge beyond the image to answer the question. Early studies retrieve required knowledge from explicit knowledge bases (KBs), which often introduces irrelevant information to the question, hence restricting the performance of their models. Recent works have sought to use a large language model (i.e., GPT-3 [3]) as an implicit knowledge engine to acquire the necessary knowledge for answering. Despite the encouraging results achieved by these methods, we argue that they have not fully activated the capacity of GPT-3 as the provided input information is insufficient. In this paper, we present Prophet-a conceptually simple framework designed to…

• NN
  National Natural Science Foundation of China

  Award: 62125201
• NK
  National Key Research and Development Program of China

  Award: 2020YFB1406701
• NS
  Natural Science Foundation of Zhejiang Province

  Award: 62072147,62020106007,61836002,LR22F020001,DT23F020007