Low-latency automotive vision with event cameras

• Published in: Nature (London) Vol. 629; no. 8014; pp. 1034 - 1040
• Main Authors: Gehrig, Daniel, Scaramuzza, Davide
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 30.05.2024; Nature Publishing Group
• Subjects: 639/166; 639/705/117; Accuracy; Advanced driver assistance systems; Algorithms; Bandwidths; Cameras; Computer vision; Efficiency; Frames per second; Graphs; Humanities and Social Sciences; Latency; Methods; multidisciplinary; Neural networks; Science; Science (multidisciplinary); Sensors; Sparsity; Temporal resolution; Tradeoffs
• ISSN: 0028-0836; 1476-4687; 1476-4687
• DOI: 10.1038/s41586-024-07409-w

The computer vision algorithms used currently in advanced driver assistance systems rely on image-based RGB cameras, leading to a critical bandwidth–latency trade-off for delivering safe driving experiences. To address this, event cameras have emerged as alternative vision sensors. Event cameras measure the changes in intensity asynchronously, offering high temporal resolution and sparsity, markedly reducing bandwidth and latency requirements 1 . Despite these advantages, event-camera-based algorithms are either highly efficient but lag behind image-based ones in terms of accuracy or sacrifice the sparsity and efficiency of events to achieve comparable results. To overcome this, here we propose a hybrid event- and frame-based object detector that preserves the advantages of each modality and thus does not suffer from this trade-off. Our method exploits the high temporal resolution and sparsity of events and the rich but low temporal resolution information in standard images to generate efficient, high-rate object detections, reducing perceptual and computational latency. We show that the use of a 20 frames per second (fps) RGB camera plus an event camera can achieve the same latency as a 5,000-fps camera with the bandwidth of a 45-fps camera without compromising accuracy. Our approach paves the way for efficient and robust perception in edge-case scenarios by uncovering the potential of event cameras 2 . Use of a 20 frames per second (fps) RGB camera plus an event camera can achieve the same latency as a 5,000-fps camera with the bandwidth of a 45-fps camera without compromising accuracy.