Audio-visual speech enhancement using deep neural networks - Jen-Cheng Hou; Syu-Siang Wang; Ying-Hui Lai; Jen-Chun Lin; Yu Tsao; Hsiu-Wen Chang; Hsin-Min Wang (2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2016)
This paper proposes a novel framework that integrates audio and visual information for speech enhancement. Most speech enhancement approaches consider audio features only to design filters or transfer functions to convert noisy speech signals to clean ones. Visual data, which provide useful complementary information to audio data, have been integrated with audio data in many speech-related approaches to attain more effective speech processing performance. This paper presents our investigation into the use of the visual features of the motion of lips as additional visual information to improve the speech enhancement capability of deep neural network (DNN) speech enhancement performance. The experimental results show that the performance of DNN with audio-visual inputs exceeds that of DNN with audio inputs only in four standardized objective evaluations, thereby confirming the effectiveness of the inclusion of visual information into an audio-only speech enhancement framework.