Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis

• Published in: Proceedings / IEEE International Conference on Computer Vision pp. 286 - 295
• Main Authors: Singh, Nikhil, Mentch, Jeff, Ng, Jerry, Beveridge, Matthew, Drori, Iddo
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2021
• Subjects: Convolution; Games; Music; Neural networks; Production; Teleconferencing; Vision + other modalities; Vision applications and systems
• ISSN: 2380-7504
• DOI: 10.1109/ICCV48922.2021.00035

Measuring the acoustic characteristics of a space is often done by capturing its impulse response (IR), a representation of how a full-range stimulus sound excites it. This work generates an IR from a single image, which can then be applied to other signals using convolution, simulating the reverberant characteristics of the space shown in the image. Recording these IRs is both time-intensive and expensive, and often infeasible for inaccessible locations. We use an end-to-end neural network architecture to generate plausible audio impulse responses from single images of acoustic environments. We evaluate our method both by comparisons to ground truth data and by human expert evaluation. We demonstrate our approach by generating plausible impulse responses from diverse settings and formats including well known places, musical halls, rooms in paintings, images from animations and computer games, synthetic environments generated from text, panoramic images, and video conference backgrounds.