Employ AI to Improve AI Services : Q-Learning Based Holistic Traffic Control for Distributed Co-Inference in Deep Learning

• Published in: IEEE transactions on services computing Vol. 15; no. 2; pp. 627 - 639
• Main Authors: Zhang, Chaofeng, Dong, Mianxiong, Ota, Kaoru
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.03.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial intelligence; Artificial neural networks; Cloud computing; Computer networks; Deep learning; Feature extraction; Inference; Internet of Things; Machine learning; network management; Network management systems; Optimization; resource management; service-oriented systems; Stability analysis; Task analysis; Traffic control
• ISSN: 1939-1374; 2372-0204
• DOI: 10.1109/TSC.2021.3113184

As the inevitable part of intelligent service in the new era, the services for AI tasks themselves have received significant attention, which due to the urgency of energy and computing resources, is difficult to implement in a stable and widely distributed system and coordinately utilize remote edge devices and cloud. In this article, we introduce an AI-based holistic network optimization solution to schedule AI services. Our proposed deep Q-learning algorithm optimizes the overall throughput of AI co-inference tasks themselves by balancing the uneven computation resources and traffic conditions. We use a multi-hop DAG (Directed Acyclic Graph) to describe a deep neural network (DNN) based co-inference network structure and introduce the virtual queue to analyze the Lyapunov stability for the system. Then, a priority-based data forwarding strategy is proposed to maximize the bandwidth efficiency, and we develop a Real-time Deep Q-learning based Edge Forwarding Scheme Optimization Algorithm (RDFO) to maximize the overall task processing rate. Finally, we conduct the platform simulation for the distributed co-inference system. Through the comparison with other benchmarks, we testify to the optimality of our proposal.