Automated Design of Metaheuristics Using Reinforcement Learning Within a Novel General Search Framework

• Published in: IEEE transactions on evolutionary computation Vol. 27; no. 4; pp. 1072 - 1084
• Main Authors: Yi, Wenjie, Qu, Rong, Jiao, Licheng, Niu, Ben
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; Algorithms; Automated algorithm design; Automation; Combinatorial analysis; Evolutionary computation; general search framework; Heuristic algorithms; Heuristic methods; Machine learning; Machine learning algorithms; metaheuristic; Metaheuristics; Optimization; Reinforcement learning; Search problems; Vehicle routing; vehicle routing problem
• ISSN: 1089-778X; 1941-0026; 1941-0026
• DOI: 10.1109/TEVC.2022.3197298

Metaheuristic algorithms have been investigated intensively to address highly complex combinatorial optimization problems. However, most metaheuristic algorithms have been designed manually by researchers of different expertise without a consistent framework. This article proposes a general search framework (GSF) to formulate in a unified way a range of different metaheuristics. With generic algorithmic components, including selection heuristics and evolution operators, the unified GSF aims to serve as the basis of analyzing algorithmic components for automated algorithm design. With the established new GSF, two reinforcement learning (RL)-based methods, deep <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula>-network based and proximal policy optimization-based methods, have been developed to automatically design a new general population-based algorithm. The proposed RL-based methods are able to intelligently select and combine appropriate algorithmic components during different stages of the optimization process. The effectiveness and generalization of the proposed RL-based methods are validated comprehensively across different benchmark instances of the capacitated vehicle routing problem with time windows. This study contributes to making a key step toward automated algorithm design with a general framework supporting fundamental analysis by effective machine learning.