A Knee-Guided Evolutionary Algorithm for Compressing Deep Neural Networks

• Published in: IEEE transactions on cybernetics Vol. 51; no. 3; pp. 1626 - 1638
• Main Authors: Zhou, Yao, Yen, Gary G., Yi, Zhang
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Compressing; Computational modeling; Deep neural networks (DNNs); Degradation; evolutionary algorithm; Evolutionary algorithms; Evolutionary computation; filter pruning; Genetic algorithms; Knee; knee solution; multiobjective optimization; Multiple objective analysis; Neural networks; Optimization; Parameter estimation; Performance degradation; Performance evaluation; Redundancy; Searching; Tradeoffs
• ISSN: 2168-2267; 2168-2275; 2168-2275
• DOI: 10.1109/TCYB.2019.2928174

Deep neural networks (DNNs) have been regarded as fundamental tools for many disciplines. Meanwhile, they are known for their large-scale parameters, high redundancy in weights, and extensive computing resource consumptions, which pose a tremendous challenge to the deployment in real-time applications or on resource-constrained devices. To cope with this issue, compressing DNNs for accelerating its inference has drawn extensive interest recently. The basic idea is to prune parameters with little performance degradation. However, the overparameterized nature and the conflict between parameters reduction and performance maintenance make it prohibitive to manually search the pruning parameter space. In this paper, we formally establish filter pruning as a multiobjective optimization problem, and propose a knee-guided evolutionary algorithm (KGEA) that can automatically search for the solution with quality tradeoff between the scale of parameters and performance, in which both conflicting objectives can be optimized simultaneously. In particular, by incorporating a minimum Manhattan distance approach, the search effort in the proposed KGEA is explicitly guided toward the knee area, which greatly facilitates the manual search for a good tradeoff solution. Moreover, the parameter importance is directly estimated on the criterion of performance loss, which can robustly identify the redundancy. In addition to the knee solution, a performance-improved model can also be found in a fine-tuning-free fashion. The experiments on compressing fully convolutional LeNet and VGG-19 networks validate the superiority of the proposed algorithm over the state-of-the-art competing methods.