Optimize TSK Fuzzy Systems for Regression Problems: Minibatch Gradient Descent With Regularization, DropRule, and AdaBound (MBGD-RDA)

• Published in: IEEE transactions on fuzzy systems Vol. 28; no. 5; pp. 1003 - 1015
• Main Authors: Wu, Dongrui, Yuan, Ye, Huang, Jian, Tan, Yihua
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: AdaBound; Adaptive-network-based fuzzy inference system (ANFIS); Algorithms; Artificial neural networks; Biological neural networks; Computational modeling; Datasets; DropRule; Fuzzy logic; Fuzzy systems; Machine learning; Machine learning algorithms; minibatch gradient descent (MBGD); Neural networks; Optimization; Optimization techniques; Regularization; Stochastic processes; Training
• ISSN: 1063-6706; 1941-0034
• DOI: 10.1109/TFUZZ.2019.2958559

Takagi-Sugeno-Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., minibatch gradient descent (MBGD), regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, MBGD with regularization, DropRule, and AdaBound, can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, on which currently no other efficient training algorithms exist.