Video Superresolution via Motion Compensation and Deep Residual Learning

• Published in: IEEE transactions on computational imaging Vol. 3; no. 4; pp. 749 - 762
• Main Authors: Li, Dingyi, Wang, Zengfu
• Format: Journal Article
• Language: English
• Published: IEEE 01.12.2017
• Subjects: Convolutional neural networks (CNNs); deep residual learning; Image reconstruction; Image resolution; Motion compensation; Motion estimation; multi-frame super-resolution; Neural networks; Optical imaging; video super-resolution
• ISSN: 2573-0436; 2333-9403; 2333-9403
• DOI: 10.1109/TCI.2017.2671360

Video superresolution (SR) techniques are of essential usages for high-resolution display devices due to the current lack of high-resolution videos. Although many algorithms have been proposed, video SR still remains a very challenging inverse problem under different conditions. In this paper, we propose a new method for video SR named motion compensation and residual net (MCResNet). We use optical flow algorithm for motion estimation and motion compensation as a preprocessing step. Then, we employ a novel deep residual convolutional neural network (CNN) to predict a high-resolution image using multiple motion compensated observations. The new residual CNN model preserves the low-frequency contents and facilitates the restoration of high-frequency details. Our method is able to handle large and complex motions adaptively. Extensive experimental results validate that our proposed method outperforms state-of-the-art single-image-based and multi-frame-based algorithms for video SR quantitatively and qualitatively.