Comparing large Language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm
RT-RK Institute for Computer Based Systems (Serbia) · University of Belgrade · +8 more institutions
Abstract
In the era of rapid digital communication, vast amounts of textual data are generated daily, demanding efficient methods for latent content analysis to extract meaningful insights. Large Language Models (LLMs) offer potential for automating this process, yet comprehensive assessments comparing their performance to human annotators across multiple dimensions are lacking. This study evaluates the inter-rater reliability, consistency, and quality of seven state-of-the-art LLMs. These include variants of OpenAI's GPT-4, Gemini, Llama-3.1-70B, and Mixtral 8 × 7B. Their performance is compared to human annotators in analyzing sentiment, political leaning, emotional intensity, and sarcasm detection. The study…
Citation impact
- FWCI
- 95.29
- Percentile
- 100%
- References
- 43
Authors
7- LBLjubiša BojićCorresponding
RT-RK Institute for Computer Based Systems (Serbia), University of Belgrade
- OZOlga Zagovora
University of Koblenz and Landau, German Research Centre for Artificial Intelligence
- AZAsta Zelenkauskaitė
Vilnius Gediminas Technical University, Drexel University
- VVVuk Vuković
University of Montenegro
- MČMilan Čabarkapa
University of Kragujevac
Topics & keywords
- Sarcasm
- Sentiment analysis
- Politics
- Computer science
- Natural language processing
- Content (measure theory)
- Artificial intelligence
- Intensity (physics)