The Multidimensional Wisdom of Crowds
California Institute of Technology · University of California San Diego
Abstract
Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images…
Citation impact
- FWCI
- 73.17
- Percentile
- 100%
- References
- 14
Authors
4Topics & keywords
- Computer science
- Crowds
- Artificial intelligence
- Ground truth
- Annotation
- Image (mathematics)
- Class (philosophy)
- Set (abstract data type)
- Quality Education