RAPIDO: a rejuvenating adaptive PID-type optimiser for deep neural networks

• Published in: Electronics letters Vol. 55; no. 16; pp. 899 - 901
• Main Authors: Kim, S, Park, D.J, Chang, D.E
• Format: Journal Article
• Language: English
• Published: The Institution of Engineering and Technology 08.08.2019
• Subjects: activation functions; deep learning; deep neural networks; gradient descent algorithm; gradient methods; hyperbolic tangent functions; Information and communications; learning (artificial intelligence); MNIST classification task; neural nets; optimisation; pattern classification; PID controller; quadratic objective function; RAPIDO; rejuvenating adaptive PID‐type optimiser; ReLU functions; sigmoid functions; three‐term control; activation functions; pattern classification; three-term control; PID controller; sigmoid functions; ReLU functions; gradient descent algorithm; hyperbolic tangent functions; rejuvenating adaptive PID-type optimiser; deep learning; optimisation; RAPIDO; MNIST classification task; quadratic objective function; learning (artificial intelligence); gradient methods; neural nets; deep neural networks
• ISSN: 0013-5194; 1350-911X; 1350-911X
• DOI: 10.1049/el.2019.1593

The authors present a novel gradient descent algorithm called RAPIDO for deep learning. It adapts over time and performs optimisation using current, past and future information similar to the PID controller. The proposed method is suited for optimising deep neural networks that consist of activation functions such as sigmoid, hyperbolic tangent and ReLU functions because it can adapt appropriately to sudden changes in gradients. They experimentally study the authors' method and show the performance results by comparing with other methods on the quadratic objective function and the MNIST classification task. The proposed method shows better performance than the other methods.