Audio-visual speech enhancement using deep neural networks

• Published in: 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) pp. 1 - 6
• Main Authors: Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Jen-Chun Lin, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang
• Format: Conference Proceeding
• Language: English
• Published: Asia Pacific Signal and Information Processing Association 01.12.2016
• Subjects: Feature extraction; Noise measurement; Signal to noise ratio; Speech; Speech enhancement; Training; Visualization
• DOI: 10.1109/APSIPA.2016.7820732

This paper proposes a novel framework that integrates audio and visual information for speech enhancement. Most speech enhancement approaches consider audio features only to design filters or transfer functions to convert noisy speech signals to clean ones. Visual data, which provide useful complementary information to audio data, have been integrated with audio data in many speech-related approaches to attain more effective speech processing performance. This paper presents our investigation into the use of the visual features of the motion of lips as additional visual information to improve the speech enhancement capability of deep neural network (DNN) speech enhancement performance. The experimental results show that the performance of DNN with audio-visual inputs exceeds that of DNN with audio inputs only in four standardized objective evaluations, thereby confirming the effectiveness of the inclusion of visual information into an audio-only speech enhancement framework.