A survey on multimodal large language models

Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of the MLLM, such as writing stories based on images and optical character recognition-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even outperform GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First, we…

• NN
  National Natural Science Foundation of China

  Awards: 61727809, 62222213, 62406264, U22B2059, U23A20319, 62072423
• NS
  Natural Science Foundation of Sichuan Province

  Award: 2023NSFSC1402
• YS
  Young Scientists Fund