Covariance-regularized regression and classification for high dimensional problems

• Published in: Journal of the Royal Statistical Society. Series B, Statistical methodology Vol. 71; no. 3; pp. 615 - 636
• Main Authors: Witten, Daniela M., Tibshirani, Robert
• Format: Journal Article
• Language: English
• Published: Oxford, UK Oxford, UK : Blackwell Publishing Ltd 01.06.2009; Blackwell Publishing Ltd; Blackwell Publishing; Blackwell; Royal Statistical Society; Oxford University Press
• Series: Journal of the Royal Statistical Society Series B
• Subjects: Classification; Coefficients; Covariance; Covariance matrices; Covariance regularization; data collection; Datasets; Discriminant analysis; Estimating techniques; Estimation; Estimation methods; Estimators; Exact sciences and technology; Gene expression; General topics; Generalized linear models; Least squares; Linear analysis; Linear inference, regression; Linear models; Linear regression; Mathematical procedures; Mathematics; Modeling; Multivariate analysis; n[double less-than sign]p; n≪p; prediction; Probability and statistics; Probability theory and stochastic processes; Regression; Regression analysis; Regression coefficients; Regulation; Sciences and techniques of general use; Statistical methods; Statistics; Stochastic processes; Studies; Variable selection; Ridge regression; Rank statistic; Generalized linear model; Covariance regularization; Prediction theory; Multivariate analysis; Stochastic process; Linear model; Penalty method; Parametric method; Covariance; Survival function; Inverse matrix; n « p; Log likelihood; Classification; Ill posed problem; Variable selection; Censored data; Discriminant analysis; Numerical linear algebra; Prediction; Regression; Statistical estimation; Covariance matrix; Survival; Regularization method; Statistical method; Statistical regression; Regression coefficient; Selection problem; Numerical analysis; Filtering theory; Regularization
• ISSN: 1369-7412; 1467-9868; 1467-9868
• DOI: 10.1111/j.1467-9868.2009.00699.x

We propose covariance-regularized regression, a family of methods for prediction in high dimensional settings that uses a shrunken estimate of the inverse covariance matrix of the features to achieve superior prediction. An estimate of the inverse covariance matrix is obtained by maximizing the log-likelihood of the data, under a multivariate normal model, subject to a penalty; it is then used to estimate coefficients for the regression of the response onto the features. We show that ridge regression, the lasso and the elastic net are special cases of covariance-regularized regression, and we demonstrate that certain previously unexplored forms of covariance-regularized regression can outperform existing methods in a range of situations. The covariance-regularized regression framework is extended to generalized linear models and linear discriminant analysis, and is used to analyse gene expression data sets with multiple class and survival outcomes.