Video Super-Resolution Using Simultaneous Motion and Intensity Calculations

• Published in: IEEE transactions on image processing Vol. 20; no. 7; pp. 1870 - 1884
• Main Authors: Keller, S H, Lauze, F, Nielsen, M
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.07.2011; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Byproducts; Calculus of variations; Dealing; Detection, estimation, filtering, equalization, prediction; Exact sciences and technology; Image processing; Image sequences; Information, signal and communications theory; Interpolation; Mathematical analysis; Motion compensation; motion super-resolution; Optical imaging; Partial differential equations; partial differential equations (PDEs); Signal and communications theory; Signal processing; Signal resolution; Signal, noise; Spatial resolution; Spatiotemporal phenomena; Studies; super-resolution; Telecommunications and information theory; variational methods; video processing; video upscaling; Performance evaluation; video upscaling; Parameter estimation; High resolution; Upconversion; Motion estimation; Variational methods; Image processing; Superresolution; motion super-resolution; super-resolution; Signal estimation; Displacement field; Algorithm; Flow field; Partial differential equation; Video signal processing; video processing; Joint detection; Image sequence; partial differential equations (PDEs); Signal processing; Motion compensation; Optical flow
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2011.2106793

In this paper, we propose an energy-based algorithm for motion-compensated video super-resolution (VSR) targeted on upscaling of standard definition (SD) video to high-definition (HD) video. Since the motion (flow field) of the image sequence is generally unknown, we introduce a formulation for the joint estimation of a super-resolution (SR) sequence and its flow field. Via the calculus of variations, this leads to a coupled system of partial differential equations for image sequence and motion estimation. We solve a simplified form of this system and, as a by-product, we indeed provide a motion field for super-resolved sequences. To the best of our knowledge, computing super-resolved flows has not been done before. Most advanced SR methods found in literature cannot be applied to general video with arbitrary scene content and/or arbitrary optical flows, as it is possible with our simultaneous VSR method. A series of experiments shows that our method outperforms other VSR methods when dealing with general video input and that it continues to provide good results even for large scaling factors up to 8 × 8.