Foundation models in robotics: Applications, challenges, and the future

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning,…

• DF
  Directorate for Engineering

  Award: 2044149
• DA
  Defense Advanced Research Projects Agency

  Award: HR001120C0107
• OO
  Office of Naval Research

  Award: N00014-23-1-2148