Unified Language Model Pre-training for Natural Language Understanding\n and Generation

This paper presents a new Unified pre-trained Language Model (UniLM) that can\nbe fine-tuned for both natural language understanding and generation tasks. The\nmodel is pre-trained using three types of language modeling tasks:\nunidirectional, bidirectional, and sequence-to-sequence prediction. The unified\nmodeling is achieved by employing a shared Transformer network and utilizing\nspecific self-attention masks to control what context the prediction conditions\non. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0\nand CoQA question answering tasks. Moreover, UniLM achieves new\nstate-of-the-art results on five natural language generation datasets,\nincluding improving the…