The second–derivative lower–bound function (SeLF) algorithm and three acceleration techniques for maximization with strongly stable convergence

• Published in: Statistics and computing Vol. 35; no. 4
• Main Authors: Tian, Guo-Liang, Zhou, Hua, Lange, Kenneth, Li, Xun-Jian
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2025; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Computer Science; Convergence; Maximization; Maximum likelihood estimates; Optimization; Original Paper; Parameters; Probability and Statistics in Computer Science; Statistical Theory and Methods; Statistics and Computing/Statistics Programs; Acceleration technique; Strongly stable convergence; SeLF algorithm; Weakly stable convergence; MM algorithms; Convergence analysis
• ISSN: 0960-3174; 1573-1375
• DOI: 10.1007/s11222-025-10639-1

This paper proposes a new maximization method, called as the second–derivative lower–bound function (SeLF) algorithm, which is a general principle for iteratively calculating the maximum likelihood estimate (MLE) θ ^ of the parameter θ in a one–dimensional target function ℓ ( θ ) (usually, the log-likelihood or marginal log-likelihood for multi-parameter cases), and its each iteration consists of two steps: A second–derivative lower–bound function step ( SeLF-step ) and a maximization step ( M-step ). The SeLF-step finds a function b ( θ ) satisfying ℓ ′ ′ ( θ ) ⩾ b ( θ ) and constructs a surrogate function Q ( θ | θ ( t ) ) [whose form depends on θ ( t ) being the t -th iteration of θ ^ ] minorizing ℓ ( θ ) at θ = θ ( t ) . The M-step calculates the maximizer θ ( t + 1 ) of the Q ( θ | θ ( t ) ) function, which is equivalent to solving the equation ℓ ′ ( θ ) + ∫ θ ( t ) θ b ( z ) d z = 0 to obtain its explicit solution θ ( t + 1 ) . The SeLF algorithm holds two major advantages: (i) it strongly stably converges to the MLE θ ^ , in contrast to the weakly stable convergence of minorization–maximization (MM) algorithms; and (ii) it converges regardless of initial values, in contrast to Newton’s method. The key for designing the SeLF algorithm is to find a function b ( θ ) satisfying ℓ ′ ′ ( θ ) ⩾ b ( θ ) for all θ in the domain such that an explicit solution to the equation ℓ ′ ( θ ) + ∫ θ ( t ) θ b ( z ) d z = 0 is available. Furthermore, we develop three acceleration techniques, namely optimal SeLF, sub-optimal SeLF, and fast–SeLF algorithms, for the SeLF algorithm, resulting in a weakly stable convergence. Various applications in statistics of the proposed SeLF algorithm and three accelerated versions are introduced. The Dirichlet distribution illustrates the potential generalization of the SeLF algorithm to the multi-dimensional case. We study the convergence rates of these algorithms, accompanied by numerical experiments and comparisons.