Evaluating Performance of Adam Optimization by Proposing Energy Index - Bhandari, Mohan; Parajuli, Pramod; Chapagain, Pralhad; Gaur, Loveleen (Recent Trends in Image Processing and Pattern Recognition, 2022)
The adjustment of learning rate (η $$η $$ ), bias and additional parameters throughout back propagation are crucial for the performance of machine learning algorithms. Regarding optimization for algorithms, adam optimization technique tune the learning parameters by utilising the exponential decay of past gradients and their squares. However, the optimizer requires the engagement of frequently occurring features from the datasets that play a significant role for performance improvement in machine learning algorithms. In this paper, the energy model of a neuron is designed to calculate the energy index from frequently occurring features and introduced in adam optimizer. The classification performance of the proposed energy modeled adam optimizer is experimented on Logistic Regression (single layered) and Support Vector Machine (hyperplane based) machine learning algorithms utlising CIFAR10, MNIST and Fashion MNIST datasets. Optimized with proposed optimizer, Logistic Regression achieved training accuracy of 90.79%, 99.02% and 95.87% whereas Support Vector Machine achieved training accuracy of 39.04%, 80.80% and 82.29% for CIFAR10, MNIST and Fashion MNIST datasets respectively.