Artificial Intelligence Generated Synthetic Datasets as the Remedy for Data Scarcity in Water Quality Index Estimation

• Published in: Water resources management Vol. 37; no. 15; pp. 6183 - 6198
• Main Authors: Chia, Min Yan, Koo, Chai Hoon, Huang, Yuk Feng, Di Chan, Wei, Pang, Jia Yin
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01.12.2023; Springer Nature B.V
• Subjects: Artificial intelligence; Artificial neural networks; Atmospheric Sciences; Back propagation networks; Civil Engineering; Computation; Data collection; Datasets; Earth and Environmental Science; Earth Sciences; Environment; Generative adversarial networks; Geotechnical Engineering & Applied Earth Sciences; Hydrogeology; Hydrology/Water Resources; infrastructure; Learning algorithms; Machine learning; Malaysia; Neural networks; Rivers; statistics; Synthetic data; water; Water intakes; Water quality; Water resources; Water scarcity; Selangor River; Malaysia; water quality index; synthetic data; artificial intelligence; back-propagation neural network
• ISSN: 0920-4741; 1573-1650
• DOI: 10.1007/s11269-023-03650-6

Water quality index (WQI) has been utilised in many countries and regions as a numeric representation of the condition of water resources. However, the computation of the WQI involves a host of water quality variables. Although machine learning models are proven to be a promising tool to estimate WQI with lesser inputs, sufficient data or samples must be collected so that the machine learning models can be trained well. This exhibits a great challenge in places where there has been a lack of data collection infrastructure to meet the needs of machine learning models. Data scarcity is a major issue to be tackled. This study covered two major rivers that served as water intakes in Peninsular Malaysia (Selangor River and Skudai River), where four synthetic data generation methods, namely the conditional tabular generative adversarial network (CTGAN), the tabular variational autoencoder (TVAE), the Gaussian copula (GC) and the copula generative adversarial network (CopulaGAN), were used to synthesise datasets based on the real dataset. By using the pairwise correlation difference (PCD), Kullback-Leibler divergence (KLD) and the Kolmogorov-Smirnov (KS) test, the best synthetic datasets were selected for the two rivers. The CopulaGAN1 and the CopulaGAN2 yielded the best small and large synthetic datasets at Selangor River, scoring the lowest PCD, KLD and KS statistics. For the Skudai River, the TVAE1 and TVAE2 were chosen. The real and synthetic datasets were used to train the back-propagation neural network (BPNN) for the WQI estimation. Based on the various evaluation metrics, it was proven that increasing the size of training data using the synthetic data method had a positive impact on the performance of the BPNN. The BPNN trained with the CopulaGAN2 (at Selangor River) and the TVAE2 (at Skudai River) yielded more accurate estimations compared to those derived from the actual and smaller datasets. Highlights Data were insufficient to train machine learning model well in developing regions. Synthetic data methods can overcome the data scarcity issue in Malaysia. CopulaGAN and TVAE outperformed other methods at Selangor River and Skudai River. BPNN trained with synthetic datasets estimated WQI with higher accuracy.