A sequential and intensive weighted language modeling scheme for multi-task learning-based natural language understanding

Suhyune Son, Seonjeong Hwang, Sohyeun Bae, Soo Jun Park, Jang Hwan Choi

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Multi-task learning (MTL) approaches are actively used for various natural language processing (NLP) tasks. The Multi-Task Deep Neural Network (MT-DNN) has contributed significantly to improving the performance of natural language understanding (NLU) tasks. However, one drawback is that confusion about the language representation of various tasks arises during the training of the MT-DNN model. Inspired by the internal-transfer weighting of MTL in medical imaging, we introduce a Sequential and IntensiveWeighted Language Modeling (SIWLM) scheme. The SIWLM consists of two stages: (1) Sequential weighted learning (SWL), which trains a model to learn entire tasks sequentially and concentrically, and (2) Intensive weighted learning (IWL), which enables the model to focus on the central task. We apply this scheme to the MT-DNN model and call this model the MTDNN-SIWLM. Our model achieves higher performance than the existing reference algorithms on six out of the eight GLUE benchmark tasks. Moreover, our model outperforms MT-DNN by 0.77 on average on the overall task. Finally, we conducted a thorough empirical investigation to determine the optimal weight for each GLUE task.

Original languageEnglish
Article number3095
JournalApplied Sciences (Switzerland)
Volume11
Issue number7
DOIs
StatePublished - 1 Apr 2021

Bibliographical note

Publisher Copyright:
© 2021 by the authors.

Keywords

  • Language modeling
  • Multi-task learning
  • Natural language understanding
  • Neural networks
  • Supervised learning

Fingerprint

Dive into the research topics of 'A sequential and intensive weighted language modeling scheme for multi-task learning-based natural language understanding'. Together they form a unique fingerprint.

Cite this