A statistical model of neural network learning via the Cramer–Rao lower bound

• Published in: Journal of the Korean Statistical Society Vol. 50; no. 3; pp. 756 - 772
• Main Authors: Kim, Tae Yoon, Park, Inho
• Format: Journal Article
• Language: English
• Published: Singapore Springer Singapore 01.09.2021; 한국통계학회
• Subjects: Applied Statistics; Bayesian Inference; Mathematics and Statistics; Research Article; Statistical Theory and Methods; Statistics; Statistics and Computing/Statistics Programs; 통계학; Validation; Cramer–Rao lower bound; 68T05; ANL algorithm; Statistical NN learning model; Statistically successful NN learning; 68Q32
• ISSN: 1226-3192; 2005-2863
• DOI: 10.1007/s42952-021-00122-8

The neural networks (NN) remain as black boxes, albeit their quite successful stories everywhere. It is mainly because they provide only the complex structure of the underlying network with a huge validation data set whenever their serendipities reveal themselves. In this paper, we propose the statistical NN learning model related to the concept of universal Turing computer for regression predictive model. Based on this model, we define ’statistically successful NN (SSNN) learning.’ This is mainly done by calculating the well-known Cramer–Rao lower bound for the averaged square error (ASE) of NN learning. Using such formal definition, we propose an ASE-based NN learning (ANL) algorithm. The ANL algorithm not only implements the Cramer–Rao lower bound successfully but also presents an effective way to figure out a complicated geometry of ASE over hyper-parameter space for NN. This enables the ANL to be free of huge validation data set. Simple numerical simulation and real data analysis are done to evaluate performance of the ANL and present how to implement it.