Audiovisual interaction with rate-varying signals

• Published in: i-Perception (London) Vol. 13; no. 6; p. 20416695221116653
• Main Authors: Yi, Long, Sekuler, Robert
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.11.2022; Sage Publications Ltd; SAGE Publishing
• Subjects: Eyes & eyesight; Visual perception; confusion matrix; temporal correlation; rate perception; audiovisual interaction; similarity space
• ISSN: 2041-6695; 2041-6695
• DOI: 10.1177/20416695221116653

A task-irrelevant, amplitude-modulating sound influences perception of a size-modulating visual stimulus. To probe the limits of this audiovisual interaction we vary the second temporal derivative of object size and of sound amplitude. In the study’s first phase subjects see a visual stimulus size-modulating with f ″ ( x ) > 0, 0, or <0, and judge each one’s rate as increasing, constant, or decreasing. Visual stimuli are accompanied by a steady, non-modulated auditory stimulus. The novel combination of multiple stimuli and multi-alternative responses allows subjects’ similarity space to be estimated from the stimulus-response confusion matrix. In the study’s second phase, rate-varying visual stimuli are presented in concert with auditory stimuli whose second derivative also varied. Subjects identified each visual stimuli as one of the three types, while trying to ignore the accompanying sound. Unlike some previous results with f ″ ( x ) fixed at 0, performance benefits relatively little when visual and auditory stimuli share the same directional change in modulation. However, performance does drop when visual and auditory stimului differ in their directions of rate change. Our task’s computational demands may make it particularly vulnerable to the effects of a dynamic task-irrelevant stimulus.