Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture’s grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less…

• AN
  Agence Nationale de la Recherche

  Awards: ANR-10-IDEX-, ANR-10-IDEX-0001-02 PSL, ANR-10-IDEX-0001-02, ANR-10-IDEX-0001, 10-IDEX-0001-02 PSL, 10-IDEX-0001-02, ANR-10-IDEX, ANR-10-LABX-0087, 10-LABX-0087 IEC, ANR-10, 10-LABX-0087, IDEX-0001-02, ANR-10-LABX-0087 IEC