Ultrafast machine vision with 2D material neural network image sensors

• Published in: Nature (London) Vol. 579; no. 7797; pp. 62 - 66
• Main Authors: Mennel, Lukas, Symonowicz, Joanna, Wachter, Stefan, Polyushkin, Dmitry K., Molina-Mendoza, Aday J., Mueller, Thomas
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 05.03.2020; Nature Publishing Group
• Subjects: 140/125; 142/126; 639/166/987; 639/624/1075/401; 639/925/918/1054; Algorithms; Artificial neural networks; Data acquisition; Energy conversion efficiency; Energy efficiency; Humanities and Social Sciences; Image acquisition; Image contrast; Image enhancement; Image processing; Innovations; Learning algorithms; Learning theory; Machine learning; Machine vision; multidisciplinary; Neural networks; Noise; Noise reduction; Science; Science (multidisciplinary); Semiconductors; Sensors; Signal processing; Synaptic strength; Two dimensional materials; Vision systems; Austria
• ISSN: 0028-0836; 1476-4687; 1476-4687
• DOI: 10.1038/s41586-020-2038-x

Machine vision technology has taken huge leaps in recent years, and is now becoming an integral part of various intelligent systems, including autonomous vehicles and robotics. Usually, visual information is captured by a frame-based camera, converted into a digital format and processed afterwards using a machine-learning algorithm such as an artificial neural network (ANN) 1 . The large amount of (mostly redundant) data passed through the entire signal chain, however, results in low frame rates and high power consumption. Various visual data preprocessing techniques have thus been developed 2 – 7 to increase the efficiency of the subsequent signal processing in an ANN. Here we demonstrate that an image sensor can itself constitute an ANN that can simultaneously sense and process optical images without latency. Our device is based on a reconfigurable two-dimensional (2D) semiconductor 8 , 9 photodiode 10 – 12 array, and the synaptic weights of the network are stored in a continuously tunable photoresponsivity matrix. We demonstrate both supervised and unsupervised learning and train the sensor to classify and encode images that are optically projected onto the chip with a throughput of 20 million bins per second. A two-dimensional semiconductor photodiode array senses and processes optical images simultaneously without latency, and is trained to classify and encode images with high throughput, acting as an artificial neural network.