An Experimental Investigation of Discourse Expectations in Neural Language Models

Eunkyung Yi, Hyowon Cho, Sanghoun Song

Research output: Contribution to journalArticlepeer-review

Abstract

The present study reports on three language processing experiments with most up-to-date neural language models from a psycholinguistic perspective. We investigated whether and how discourse expectations demonstrated in the psycholinguistics literature are manifested in neural language models, using the language models whose architectures and assumptions are considered most appropriate for the given language processing tasks. We first attempted to perform a general assessment of a neural model’s discourse expectations about story continuity or coherence (Experiment 1), based on the next sentence prediction module of the bidirectional transformer-based model BERT (Devlin et al. 2019). We also studied language models’ expectations about reference continuity in discursive contexts in both comprehension (Experiment 2) and production (Experiment 3) settings, based on so-called Implicit Causality biases. We used the unidirectional (or left-to-right) RNN-based model LSTM (Hochreiter and Schmidhuber 1997) and the transformer-based generation model GPT-2 (Radford et al. 2019), respectively. The results of the three experiments showed, first, that neural language models are highly successful in distinguishing between reasonably expected and unexpected story continuations in human communication and also that they exhibit human-like bias patterns in reference expectations in both comprehension and production contexts. The results of the present study suggest language models can closely simulate the discourse processing features observed in psycholinguistic experiments with human speakers. The results also suggest language models can, beyond simply functioning as a technology for practical purposes, serve as a useful research tool and/or object for the study of human discourse processing.

Original languageEnglish
Pages (from-to)1101-1115
Number of pages15
JournalKorean Journal of English Language and Linguistics
Volume22
DOIs
StatePublished - 2022

Keywords

  • BERT
  • coreference resolution
  • discourse expectation
  • GPT-2
  • implicit causality bias
  • LSTM
  • neural language model
  • next sentence prediction
  • surprisal

Fingerprint

Dive into the research topics of 'An Experimental Investigation of Discourse Expectations in Neural Language Models'. Together they form a unique fingerprint.

Cite this