Increasingly complex representations of natural movies across the dorsal stream are shared between subjects

• Published in: NeuroImage (Orlando, Fla.) Vol. 145; no. Pt B; pp. 329 - 336
• Main Authors: Güçlü, Umut, van Gerven, Marcel A.J.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 15.01.2017; Elsevier Limited
• Subjects: Adult; Datasets; Decoding; Deep neural network; Dorsal stream; Encoding; Humans; Hyperalignment; Image Processing, Computer-Assisted - methods; Motion Pictures; Neural networks; Neural Networks (Computer); Studies; Visual Pathways - diagnostic imaging; Visual Pathways - physiology; Visual Perception - physiology; Deep neural network; Hyperalignment; Encoding; Dorsal stream; Decoding
• ISSN: 1053-8119; 1095-9572; 1095-9572
• DOI: 10.1016/j.neuroimage.2015.12.036

Recently, deep neural networks (DNNs) have been shown to provide accurate predictions of neural responses across the ventral visual pathway. We here explore whether they also provide accurate predictions of neural responses across the dorsal visual pathway, which is thought to be devoted to motion processing and action recognition. This is achieved by training deep neural networks to recognize actions in videos and subsequently using them to predict neural responses while subjects are watching natural movies. Moreover, we explore whether dorsal stream representations are shared between subjects. In order to address this question, we examine if individual subject predictions can be made in a common representational space estimated via hyperalignment. Results show that a DNN trained for action recognition can be used to accurately predict how dorsal stream responds to natural movies, revealing a correspondence in representations of DNN layers and dorsal stream areas. It is also demonstrated that models operating in a common representational space can generalize to responses of multiple or even unseen individual subjects to novel spatio-temporal stimuli in both encoding and decoding settings, suggesting that a common representational space underlies dorsal stream responses across multiple subjects. •Deep neural network and dorsal stream representations show correspondence.•Dorsal stream representations are shared between subjects.•A common encoder can predict fMRI responses to novel stimuli for unseen subjects.•A common decoder can identify novel stimuli from fMRI responses for unseen subjects.