Objective Prediction of Next-Day’s Affect Using Multimodal Physiological and Behavioral Data: Algorithm Development and Validation Study

• Published in: JMIR formative research Vol. 7; p. e39425
• Main Authors: Jafarlou, Salar, Lai, Jocelyn, Azimi, Iman, Mousavi, Zahra, Labbaf, Sina, Jain, Ramesh C, Dutt, Nikil, Borelli, Jessica L, Rahmani, Amir
• Format: Journal Article
• Language: English
• Published: Canada JMIR Publications 15.03.2023
• Subjects: COVID-19; Emotions; Longitudinal studies; Original Paper; Sensors; United States > US; wearable devices; affective computing; mental health
• ISSN: 2561-326X; 2561-326X
• DOI: 10.2196/39425

Affective states are important aspects of healthy functioning; as such, monitoring and understanding affect is necessary for the assessment and treatment of mood-based disorders. Recent advancements in wearable technologies have increased the use of such tools in detecting and accurately estimating mental states (eg, affect, mood, and stress), offering comprehensive and continuous monitoring of individuals over time. Previous attempts to model an individual's mental state relied on subjective measurements or the inclusion of only a few objective monitoring modalities (eg, smartphones). This study aims to investigate the capacity of monitoring affect using fully objective measurements. We conducted a comparatively long-term (12-month) study with a holistic sampling of participants' moods, including 20 affective states. Longitudinal physiological data (eg, sleep and heart rate), as well as daily assessments of affect, were collected using 3 modalities (ie, smartphone, watch, and ring) from 20 college students over a year. We examined the difference between the distributions of data collected from each modality along with the differences between their rates of missingness. Out of the 20 participants, 7 provided us with 200 or more days' worth of data, and we used this for our predictive modeling setup. Distributions of positive affect (PA) and negative affect (NA) among the 7 selected participants were observed. For predictive modeling, we assessed the performance of different machine learning models, including random forests (RFs), support vector machines (SVMs), multilayer perceptron (MLP), and K-nearest neighbor (KNN). We also investigated the capability of each modality in predicting mood and the most important features of PA and NA RF models. RF was the best-performing model in our analysis and performed mood and stress (nervousness) prediction with ~81% and ~72% accuracy, respectively. PA models resulted in better performance compared to NA. The order of the most important modalities in predicting PA and NA was the smart ring, phone, and watch, respectively. SHAP (Shapley Additive Explanations) analysis showed that sleep and activity-related features were the most impactful in predicting PA and NA. Generic machine learning-based affect prediction models, trained with population data, outperform existing methods, which use the individual's historical information. Our findings indicated that our mood prediction method outperformed the existing methods. Additionally, we found that sleep and activity level were the most important features for predicting next-day PA and NA, respectively.