Generic Path Algorithm for Regularized Statistical Estimation

• Published in: Journal of the American Statistical Association Vol. 109; no. 506; pp. 686 - 699
• Main Authors: Zhou, Hua, Wu, Yichao
• Format: Journal Article
• Language: English
• Published: United States Taylor & Francis 01.06.2014; Taylor & Francis Group, LLC; Taylor & Francis Ltd
• Subjects: Algorithms; artificial intelligence; Constrained optimization; Density; Density estimation; Equality; equations; Estimating techniques; Gaussian graphical model; Generalized linear model; Generalized linear models; Graphical models; Inequality; Lasso; Learning; Least squares; Linear analysis; linear models; Linear regression; Log-concave density estimation; Mathematical vectors; Matrices; Maximum likelihood estimation; Ordinary differential equations; Penalties; Quasi-likelihoods; regression analysis; Regularization; Shape restricted regression; Solution path; Statistical estimation; Statistics; Theory and Methods; generalized linear model; log-concave density estimation; quasi-likelihoods; solution path; ordinary differential equations; lasso; regularization; Gaussian graphical model; shape restricted regression
• ISSN: 1537-274X; 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2013.864166

Regularization is widely used in statistics and machine learning to prevent overfitting and gear solution toward prior information. In general, a regularized estimation problem minimizes the sum of a loss function and a penalty term. The penalty term is usually weighted by a tuning parameter and encourages certain constraints on the parameters to be estimated. Particular choices of constraints lead to the popular lasso, fused-lasso, and other generalized ℓ ₁ penalized regression methods. In this article we follow a recent idea by Wu and propose an exact path solver based on ordinary differential equations (EPSODE) that works for any convex loss function and can deal with generalized ℓ ₁ penalties as well as more complicated regularization such as inequality constraints encountered in shape-restricted regressions and nonparametric density estimation. Nonasymptotic error bounds for the equality regularized estimates are derived. In practice, the EPSODE can be coupled with AIC, BIC, C ₚ or cross-validation to select an optimal tuning parameter, or provide a convenient model space for performing model averaging or aggregation. Our applications to generalized ℓ ₁ regularized generalized linear models, shape-restricted regressions, Gaussian graphical models, and nonparametric density estimation showcase the potential of the EPSODE algorithm. Supplementary materials for this article are available online.