Variable Inclusion and Shrinkage Algorithms

• Published in: Journal of the American Statistical Association Vol. 103; no. 483; pp. 1304 - 1315
• Main Authors: Radchenko, Peter, James, Gareth M
• Format: Journal Article
• Language: English
• Published: Alexandria, VA Taylor & Francis 01.09.2008; American Statistical Association; Taylor & Francis Ltd
• Subjects: Algorithms; Applications; Branch & bound algorithms; Coefficients; Correlations; Covariance; Dantzig selector; Datasets; Error rates; Exact sciences and technology; Feature selection; Forecasts; General topics; Generalized linear model; Generalized linear models; Justification; Lasso; Linear analysis; Linear inference, regression; Mathematical analysis; Mathematical independent variables; Mathematics; Methodology; Modeling; Parametric inference; Probability and statistics; Regression analysis; Sciences and techniques of general use; Statistical analysis; Statistical methods; Statistics; Structural analysis; Theory and Methods; Variable coefficients; Variable selection; Variables; Ridge regression; Rank statistic; Estimation error; Generalized linear model; Error estimation; Linear regression; Lasso; Shrinkage estimator; Algorithm; Linear model; Dantzig selector; Parametric method; Statistical method; Statistical regression; Regression coefficient; Selection problem; Simulation; Regression model; Law of large numbers; Application; Variable selection
• ISSN: 0162-1459; 1537-274X
• DOI: 10.1198/016214508000000481

The Lasso is a popular and computationally efficient procedure for automatically performing both variable selection and coefficient shrinkage on linear regression models. One limitation of the Lasso is that the same tuning parameter is used for both variable selection and shrinkage. As a result, it typically ends up selecting a model with too many variables to prevent overshrinkage of the regression coefficients. We suggest an improved class of methods called variable inclusion and shrinkage algorithms (VISA). Our approach is capable of selecting sparse models while avoiding overshrinkage problems and uses a path algorithm, and so also is computationally efficient. We show through extensive simulations that VISA significantly outperforms the Lasso and also provides improvements over more recent procedures, such as the Dantzig selector, relaxed Lasso, and adaptive Lasso. In addition, we provide theoretical justification for VISA in terms of nonasymptotic bounds on the estimation error that suggest it should exhibit good performance even for large numbers of predictors. Finally, we extend the VISA methodology, path algorithm, and theoretical bounds to the generalized linear models framework.