Accelerating on-device DNN inference during service outage through scheduling early exit

• Published in: Computer communications Vol. 162; pp. 69 - 82
• Main Authors: Wang, Zizhao, Bao, Wei, Yuan, Dong, Ge, Liming, Tran, Nguyen H., Zomaya, Albert Y.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.10.2020
• Subjects: Computation offloading; DNN inference; Early exit; Edge computing; DNN inference; Computation offloading; Early exit; Edge computing
• ISSN: 0140-3664; 1873-703X
• DOI: 10.1016/j.comcom.2020.08.005

In recent years, the rapid development of edge computing enables us to process a wide variety of intelligent applications at the edge, such as real-time video analytics. However, edge computing could suffer from service outage caused by the fluctuated wireless connection or congested computing resource. During the service outage, the only choice is to process the deep neural network (DNN) inference at the local mobile devices. The obstacle is that due to the limited resource, it may not be possible to complete inference tasks on time. Inspired by the recently developed early exit of DNNs, where we can exit DNN at earlier layers to shorten the inference delay by sacrificing an acceptable level of accuracy, we propose to adopt such mechanism to process inference tasks during the service outage. The challenge is how to obtain the optimal schedule with diverse early exit choices. To this end, we formulate an optimal scheduling problem with the objective to maximize a general overall utility. However, the problem is in the form of integer programming, which cannot be solved by a standard approach. We therefore prove the Ordered Scheduling structure, indicating that a frame arrived earlier must be scheduled earlier. Such structure greatly decreases the searching space for an optimal solution. Then, we propose the Scheduling Early Exit (SEE) algorithm based on dynamic programming, to solve the problem optimally with polynomial computational complexity. Finally, we conduct trace-driven simulations and real-world experiment to compare SEE with two benchmarks. The result shows that the utility gain of SEE can outperform the benchmarks by 50.9% in the simulation and by 57.79% in the real-world experiment.