A new approach to variable selection in least squares problems

• Published in: IMA journal of numerical analysis Vol. 20; no. 3; pp. 389 - 403
• Main Authors: Osborne, MR, Presnell, B, Turlach, BA
• Format: Journal Article
• Language: English
• Published: Oxford Oxford University Press 01.07.2000; Oxford Publishing Limited (England)
• Subjects: Calculus of variations and optimal control; Exact sciences and technology; Experimental design; Mathematical analysis; Mathematics; Probability and statistics; Sciences and techniques of general use; Statistics; Unconstrained optimization; Least squares method; Gram Schmidt orthogonalization; Homotopy; Descent method; Lasso method; Scaling; Implementation; Constrained optimization; Variable selection
• ISSN: 0272-4979; 1464-3642
• DOI: 10.1093/imanum/20.3.389

The title Lasso has been suggested by Tibshirani (1996) as a colourful name for a technique of variable selection which requires the minimization of a sum of squares subject to an l1 bound κ on the solution. This forces zero components in the minimizing solution for small values of κ. Thus this bound can function as a selection parameter. This paper makes two contributions to computational problems associated with implementing the Lasso: (1) a compact descent method for solving the constrained problem for a particular value of κ is formulated, and (2) a homotopy method, in which the constraint bound κ becomes the homotopy parameter, is developed to completely describe the possible selection regimes. Both algorithms have a finite termination property. It is suggested that modified Gram-Schmidt orthogonalization applied to an augmented design matrix provides an effective basis for implementing the algorithms.