Evidence of a predictive coding hierarchy in the human brain listening to speech

• Published in: Nature human behaviour Vol. 7; no. 3; pp. 430 - 441
• Main Authors: Caucheteux, Charlotte, Gramfort, Alexandre, King, Jean-Rémi
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 01.03.2023; Nature Publishing Group; Nature Research [2017-....]
• Subjects: 631/378/116; 631/378/2649/1594; 639/705/117; Algorithms; Artificial Intelligence; Auditory Perception - physiology; Behavioral Sciences; Biomedical and Life Sciences; Brain; Brain - diagnostic imaging; Brain - physiology; Cognition; Cognitive science; Computer Science; Deep learning; Experimental Psychology; Functional magnetic resonance imaging; Humans; Language; Language modeling; Language thought relationship; Life Sciences; Listening; Machine learning; Mapping; Microeconomics; Natural language processing; Neurosciences; Personality and Social Psychology; Short stories; Speech; Speech - physiology; Temporal Lobe - physiology; Neuroscience; Predictive Coding; Deep Learning; Natural Language Processing
• ISSN: 2397-3374; 2397-3374
• DOI: 10.1038/s41562-022-01516-2

Considerable progress has recently been made in natural language processing: deep learning algorithms are increasingly able to generate, summarize, translate and classify texts. Yet, these language models still fail to match the language abilities of humans. Predictive coding theory offers a tentative explanation to this discrepancy: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition. Current machine learning language algorithms make adjacent word-level predictions. In this work, Caucheteux et al. show that the human brain probably uses long-range and hierarchical predictions, taking into account up to eight possible words into the future.