GAN-Based Day-to-Night Image Style Transfer for Nighttime Vehicle Detection

• Published in: IEEE transactions on intelligent transportation systems Vol. 22; no. 2; pp. 951 - 963
• Main Authors: Lin, Che-Tsung, Huang, Sheng-Wei, Wu, Yen-Yi, Lai, Shang-Hong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Detectors; domain adaptation; Domains; Feature extraction; Gallium nitride; generative adversarial network; Image processing; Image segmentation; image-to-image translation; Model accuracy; Night; Object detection; Object recognition; semantic segmentation; Training; Vehicle detection; Vehicle detectors
• ISSN: 1524-9050; 1558-0016
• DOI: 10.1109/TITS.2019.2961679

Data augmentation plays a crucial role in training a CNN-based detector. Most previous approaches were based on using a combination of general image-processing operations and could only produce limited plausible image variations. Recently, GAN (Generative Adversarial Network) based methods have shown compelling visual results. However, they are prone to fail at preserving image-objects and maintaining translation consistency when faced with large and complex domain shifts, such as day-to-night. In this paper, we propose AugGAN, a GAN-based data augmenter which could transform on-road driving images to a desired domain while image-objects would be well-preserved. The contribution of this work is three-fold: (1) we design a structure-aware unpaired image-to-image translation network which learns the latent data transformation across different domains while artifacts in the transformed images are greatly reduced; (2) we quantitatively prove that the domain adaptation capability of a vehicle detector is not limited by its training data; (3) our object-preserving network provides significant performance gain in the difficult day-to-night case in terms of vehicle detection. AugGAN could generate more visually plausible images compared to competing methods on different on-road image translation tasks across domains. In addition, we quantitatively evaluate different methods by training Faster R-CNN and YOLO with datasets generated from the transformed results and demonstrate significant improvement on the object detection accuracies by using the proposed AugGAN model.