Pre-Trained Contextualized Word Representations for Extractive Text Summarization

Recent developments in natural language understanding (NLU) research
have given rise to pre-trained language models for contextualized word
representations. XLNet, the most recent iteration of these generalized
language models, advances the state-of-the-art on multiple benchmarks,
outperforming its highly successful predecessor BERT on 18 of 20
language understanding tasks. Several successful attempts at adapting
BERT for both extractive and abstractive text summarization have
recently been proposed. In this study, we leverage XLNet for the task
of extractive summarization by stacking and jointly fine-tuning a
sentence-level classification layer on output representations for
sentence selection. Results on the CNN/DailyMail text summarization
dataset are competitive with the state-of-the-art with an architecture
that has fewer trainable parameters than other approaches.