articleIEEE Transactions on Neural Networks and Learning SystemsSep 10, 2020HYBRID OA

Attention in Natural Language Processing

AGAndrea GalassiMLMarco LippiPTPaolo Torroni

University of Bologna · University of Modena and Reggio Emilia

PubMed
Indexed inarxivcrossrefpubmed

Abstract

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information…

Citation impact

625
total citations
FWCI
46.52
Percentile
100%
References
257
Citations per year

Authors

3

Topics & keywords

Topics
Keywords
  • Computer science
  • Categorization
  • Artificial intelligence
  • Variety (cybernetics)
  • Natural language
  • Natural language understanding
  • Data science
  • Natural language processing
UN Sustainable Development Goals
  • Quality Education
No related works found for this paper.

Funding