TinyBERT: Distilling BERT for Natural Language Understanding

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resourcerestricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large "teacher" BERT can be effectively transferred to a small "student" Tiny-BERT. Then, we introduce a new two-stage learning framework for…

• NN
  National Natural Science Foundation of China

  Awards: 61772216, 61821003, NO.61832020, No.61821003, 61832020, No.61821003
• NS
  National Science and Technology Major Project

  Awards: No.2017ZX01032-101, 2017ZX01032-101
• FR
  Fundamental Research Funds for the Central Universities