Sparse Algorithms Are Not Stable: A No-Free-Lunch Theorem

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 34; no. 1; pp. 187 - 193
• Main Authors: Huan Xu, Caramanis, C., Mannor, S.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.01.2012; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithmics. Computability. Computer arithmetics; Algorithms; Applied sciences; Artificial intelligence; Computer science; control theory; systems; Exact sciences and technology; Intelligence; Lasso; Learning; Machine learning algorithms; Pattern analysis; Regression; regularization; Signal processing algorithms; sparsity; Stability; Stability criteria; Support vector machines; Theorems; Theoretical computing; Vices; Stability; Least squares method; Lasso; Algorithmics; Sparse matrix; Sparse representation; sparsity; Learning algorithm; Artificial intelligence; Regularization; Constrained optimization
• ISSN: 0162-8828; 1939-3539; 2160-9292; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2011.177

We consider two desired properties of learning algorithms: sparsity and algorithmic stability. Both properties are believed to lead to good generalization ability. We show that these two properties are fundamentally at odds with each other: A sparse algorithm cannot be stable and vice versa. Thus, one has to trade off sparsity and stability in designing a learning algorithm. In particular, our general result implies that ℓ 1 -regularized regression (Lasso) cannot be stable, while ℓ 2 -regularized regression is known to have strong stability properties and is therefore not sparse.