ERNIE: Enhanced Language Representation with Informative Entities

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge…

• CA
  China Association for Science and Technology

  Award: 2016QNRC001
• DF
  Deutsche Forschungsgemeinschaft

  Awards: 61621136008, DFG TRR-169, NSFC 61621136008, TRR-169
• NN
  National Natural Science Foundation of China

  Awards: 61621136008, NSFC 61621136008, TRR-169, 2016QNRC001, 61572273