Derivative-free optimization methods

• Published in: Acta numerica Vol. 28; no. 2010; pp. 287 - 404
• Main Authors: Larson, Jeffrey, Menickelly, Matt, Wild, Stefan M.
• Format: Journal Article
• Language: English
• Published: Cambridge, UK Cambridge University Press 01.05.2019
• Subjects: Algorithms; Artificial intelligence; Bibliographic literature; Convexity; Estimates; Machine learning; Mathematical functions; MATHEMATICS AND COMPUTING; Methods; Optimization
• ISSN: 0962-4929; 1474-0508
• DOI: 10.1017/S0962492919000060

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.