Projected Newton-type Methods in Machine Learning - Mark Schmidt; Dongmin Kim; Suvrit Sra (Optimization for Machine Learning, 2011)
We study Newton-type methods for solving the optimization problem $\mathop {\min }\limits_x f(x) + r(x)$, subject to$x \in Ω $, (11.1) where$f:{R^n} → R$is twice continuously differentiable and convex;$r:{R^n} → R$is continuous and convex, but not necessarily differentiable everywhere; and$Ω $is a simple convex constraint set. This formulation is general and captures numerous problems in machine learning, especially wherefcorresponds to a loss, andrto a regularizer. Let us, however, defer concrete examples of (11.1) until we have developed some theoretical background. We propose to solve (11.1) via Newton-type methods, a certain class of second-order methods that are known to often work well for