Audio-visual spatial alignment improves integration in the presence of a competing audio-visual stimulus

• Published in: Neuropsychologia Vol. 146; p. 107530
• Main Authors: Fleming, Justin T., Noyce, Abigail L., Shinn-Cunningham, Barbara G.
• Format: Journal Article
• Language: English
• Published: England Elsevier Ltd 01.09.2020
• Subjects: Attention; Audio-visual integration; Electroencephalography; Spatial alignment; Temporal coherence; Visual search; Audio-visual integration; Electroencephalography; Spatial alignment; Visual search; Temporal coherence; Attention
• ISSN: 0028-3932; 1873-3514; 1873-3514
• DOI: 10.1016/j.neuropsychologia.2020.107530

In order to parse the world around us, we must constantly determine which sensory inputs arise from the same physical source and should therefore be perceptually integrated. Temporal coherence between auditory and visual stimuli drives audio-visual (AV) integration, but the role played by AV spatial alignment is less well understood. Here, we manipulated AV spatial alignment and collected electroencephalography (EEG) data while human subjects performed a free-field variant of the “pip and pop” AV search task. In this paradigm, visual search is aided by a spatially uninformative auditory tone, the onsets of which are synchronized to changes in the visual target. In Experiment 1, tones were either spatially aligned or spatially misaligned with the visual display. Regardless of AV spatial alignment, we replicated the key pip and pop result of improved AV search times. Mirroring the behavioral results, we found an enhancement of early event-related potentials (ERPs), particularly the auditory N1 component, in both AV conditions. We demonstrate that both top-down and bottom-up attention contribute to these N1 enhancements. In Experiment 2, we tested whether spatial alignment influences AV integration in a more challenging context with competing multisensory stimuli. An AV foil was added that visually resembled the target and was synchronized to its own stream of synchronous tones. The visual components of the AV target and AV foil occurred in opposite hemifields; the two auditory components were also in opposite hemifields and were either spatially aligned or spatially misaligned with the visual components to which they were synchronized. Search was fastest when the auditory and visual components of the AV target (and the foil) were spatially aligned. Attention modulated ERPs in both spatial conditions, but importantly, the scalp topography of early evoked responses shifted only when stimulus components were spatially aligned, signaling the recruitment of different neural generators likely related to multisensory integration. These results suggest that AV integration depends on AV spatial alignment when stimuli in both modalities compete for selective integration, a common scenario in real-world perception. •Visual search was speeded by a synchronous tone (the pip and pop effect).•AV search benefits did not require spatial alignment between the tones and target.•Neural responses to the tone were enhanced in the AV conditions.•When two AV objects were present, only spatially aligned AV search was speeded.•Early EEG response topography was shifted only in the spatially aligned condition.