Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts

Pre-trained large language models (“LLMs”) like GPT-3 can engage in fluent, multi-turn instruction-taking out-of-the-box, making them attractive materials for designing natural language interactions. Using natural language to steer LLM outputs (“prompting”) has emerged as an important design technique potentially accessible to non-AI-experts. Crafting effective prompts can be challenging, however, and prompt-based interactions are brittle. Here, we explore whether non-AI-experts can successfully engage in “end-user prompt engineering” using a design probe—a prototype LLM-based chatbot design tool supporting development and systematic evaluation of prompting strategies. Ultimately, our probe participants…

• DA
  Defense Advanced Research Projects Agency

  Awards: FA8750, FA8750-20-C-0156, FA8750-20-C-0074, and FA8750-20-C0155, FA8750-20-C-0156
• UA
  U.S. Air Force