Utilizing machine learning for detecting flowering in mid-range digital repeat photography

• Published in: Ecological modelling Vol. 440; p. 109419
• Main Authors: Kim, Tae Kyung, Kim, Sukyung, Won, Myoungsoo, Lim, Jong-Hwan, Yoon, Sukhee, Jang, Keunchang, Lee, Kye-Han, Park, Yeong Dae, Kim, Hyun Seok
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 15.01.2021
• Subjects: climate change; color; data collection; digital images; leaves; neural networks; phenology; photography; time series analysis; trees
• ISSN: 0304-3800
• DOI: 10.1016/j.ecolmodel.2020.109419

The responses of plants to climate change are typically reflected in the changes in leaf and flowering phenology. By exploiting the strength and simplicity of repeated digital photography and color indices, a majority of the phenological studies have been successful at investigating leaf phenology, while flowering phenology is rarely studied using the automatic capture and analysis of repeated photography. In this study, we trained and tested 5 different pretrained Convolutional Neural Network (CNN) algorithms to detect flowering events from images of white colored flowering trees and analyzed the possible factors that can affect the performance of the models. We collected images from the web and processed the images into a binary classification dataset in which a positive label indicated a tree in bloom. We also installed time-lapse cameras and captured images to validate the performances of the models in the real-world. Regarding the CNN architectures, the VGG16, ResNet50, ResNet101, MobileNet, and NASNet models were adopted, and the model weights were pretrained using the ImageNet-1000 dataset. After 20 epochs of training with 16,005 images, all of the models were successfully trained, reaching over 98% test accuracy, and 4 models reached over 99% test accuracy. All the models also showed accurate and stable performances in detecting flowering in time-series datasets with a minor inconstancy at the beginning of the flowering stages. Overall, the NASNet model showed the best performance in both the test dataset and the time-series datasets. A detailed analysis of the performance revealed that the models were especially prone to misclassify images with small relative flowering areas and were affected by the number of samples in the training dataset. We concluded that the preprocessing of the images and the size of the training dataset are essential for the high performance of the models compared to the architecture of the individual models. Furthermore, in addition to the need for a larger dataset, the proper resolution is required to successfully detect flowering from repeated photography, and most current phenological networks do not meet this condition. We suggest that mid-range photography combined with CNN algorithms can be a legitimate approach to properly accumulate and automatically process the data for studying flowering phenology. [Display omitted]