Advancements in Transfer Learning for Natural Language Processing Tasks

Abstract

Transfer learning is a new craze in the field of natural language processing, not just for beating task-specific performance records in many natural language processing tasks by huge computation and parameter settings with extensive training, but also with so much of model reusability. The transformer's rise, beginning with the attention is all you need paper by Vaswani et al. with the introduction of BERT transformer (Bidirectional Encoder Representations from Transformers) by the Google research team, has been spectacular. BERT has been designed to pre-train deep bidirectional representations from unlabelled text by jointly conditioning on left and right context in all layers and has subsequently paved way for many transformer-based models since November 2018.

Transfer learning is the new craze in modern natural language processing applications. Established by the concept of transfer learning, with more data and training time, much larger models have been built pre-trained on a large amount of training examples. These models are then fine-tuned with much smaller datasets for different linguistic tasks and applications. As an outcome of this modern trend, much better performance on the fine-tuning tasks has been observed. In this chapter, we look at various training paradigms of transfer learning. We explore a few fine-tuning methods and list several state-of-the-art results of different linguistic tasks. Finally, we talk about deployment in a real-world or for commercial natural language processing tasks.