Probabilistic Prediction and the Continuity of Language Comprehension

Abstract

It is well known that humans comprehend language in an incremental fashion, and that while doing so they use diverse contextual clues to make predictions about upcoming words (e.g. Tanenhaus et al., 1995). One way to model such predictions quantitatively is to use conditional probability, with P(continuation|context) denoting the fraction of the time that some continuation is expected to occur in a given context (Hale, 2001). For cognitive linguistics, such probabilities have special appeal. Generally, we seek theories that can faithfully describe the messy world of language and thought, and that are precise enough to make sharp, testable predictions; in practice we rarely achieve both goals simultaneously. Conditional probability provides a (partial) framework for representing linguistic knowledge that is subject to precise mathematics, but at the same time - unlike traditional formalisms - is amenable to incremental learning and gradient representations, and generalizes naturally to all levels of linguistic and extra-linguistic structure. Understanding the details of prediction in online language comprehension, therefore, may eventually pay considerable theoretical dividends.

To this end, consider one behavioral correlate of predictability: words which are more predictable are processed faster (e.g. Ehrlich and Rayner, 1981). This effect is well known, but there is currently no agreement on why it occurs; and, since previous studies (Rayner and Well, 1996) have used exclusively factorial comparisons, we know the effect's direction but have little insight into its functional form.

To address these issues, we first present a novel theory of linguistic processing time that draws inspiration from two sources: the literature on motor control, and on construction grammar (Fillmore, 1988; Kay and Fillmore, 1999). The model consists of an optimal control system (Todorov, 2004) that, rather than controlling muscles, controls the allocation of preparatory resources in the linguistic processing system. Its challenge is to manage the trade-off between conserving resources and processing quickly; for efficiency, it preferentially allocates resources to more probable continuations, which results in a speedup for predictable words. Next, following the principles of construction grammar, we constrain the model by requiring that it have no preferred scale - we assume that processing proceeds continuously and simultaneously at all levels from phoneme to clause, using similar mechanisms. This turns out to make a strong prediction: processing time for an item should be proportional to the logarithm of that item's probability-in-context.

Second, we test this prediction by analyzing the relation between probability and fixation time in the Dundee eye-movement corpus (Kennedy et al., 2003), approximating probability using a computational language model. As compared to previous studies, this approach has the advantage of both improved ecological validity and vastly greater statistical power, allowing the examination of curve shape. We find that after controlling for confounds, probability does have a logarithmic effect on standard reading-time measures, and this effect is substantial and systematic over several orders of magnitude.

This result invalidates a number of competing theories which predict different curve shapes, confirms the prediction of our model, and thus provides supporting evidence for constructional accounts of language processing.

Keywords: cognitive linguistics, constructions, grammar, construction grammar