The Promises and Pitfalls of Large Language Models as Feedback Providers: A Study of Prompt Engineering and the Quality of AI-Driven Feedback

To address these questions, we developed a theory-driven manual to evaluate prompt quality and designed three prompts of varying quality. Feedback generated by ChatGPT-4 was assessed alongside feedback from novices and experts, who were provided with the highest-quality prompt.

Our findings reveal that only the best prompt consistently produced high-quality feedback. Additionally, LLM feedback outperformed novice feedback and, in the categories explanation, questions, and specificity, even surpassed expert feedback in quality while being generated more quickly.