6-DOF VR videos with a single 360-camera

• Published in: Proceedings of the Workshop on Future Trends of Distributed Computing Systems pp. 37 - 44
• Main Authors: Jingwei Huang, Zhili Chen, Ceylan, Duygu, Hailin Jin
• Format: Conference Proceeding
• Language: English
• Published: IEEE 2017
• Subjects: Cameras; Geometry; Headphones; I.2.10 [Artificial Intelligence]: Vision and Scene Understanding - Video analysis; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism - Virtual reality; I.4.8 [Image Processing and Computer Vision]: Scene Analysis - Motion; Image reconstruction; Three-dimensional displays; Tracking; Videos
• ISSN: 2375-5334
• DOI: 10.1109/VR.2017.7892229

Recent breakthroughs in consumer level virtual reality (VR) headsets are creating a growing user-base in demand for immersive, full 3D VR experiences. While monoscopic 360-videos are perhaps the most prevalent type of content for VR headsets, they lack 3D information and thus cannot be viewed with full 6 degree-of-freedom (DOF). We present an approach that addresses this limitation via a novel warping algorithm that can synthesize new views both with rotational and translational motion of the viewpoint. This enables the ability to perform VR playback of input monoscopic 360-videos files in full stereo with full 6-DOF of head motion. Our method synthesizes novel views for each eye in accordance with the 6-DOF motion of the headset. Our solution tailors standard structure-from-motion and dense reconstruction algorithms to work accurately for 360-videos and is optimized for GPUs to achieve VR frame rates (>120 fps). We demonstrate the effectiveness our approach on a variety of videos with interesting content.