Reinforcement Learning-Based Switching Controller for a Milliscale Robot in a Constrained Environment

• Published in: IEEE transactions on automation science and engineering Vol. 21; no. 2; pp. 1 - 17
• Main Authors: Tariverdi, Abbas, Cote-Allard, Ulysse, Mathiassen, Kim, Elle, Ole J., Kalvoy, Havard, Martinsen, Orjan G., Torresen, Jim
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Active control; Algorithms; attractor dynamics; Computer & video games; Control systems; Control systems design; Controllers; Customization; Deep learning; Disturbances; Drugs; Endoscopes; ERRT; Feasibility studies; Ferromagnetism; Heuristic algorithms; Intestine; Kinematics; Machine learning; Magnetic manipulation; Navigation; Quantiles; Rainbows; rapidly-exploring random trees; reinforcement learning; Robot arms; Robot control; Robot learning; Robotics; Robots; Robustness; Simulation; Switches; Switching; switching control; targeted drug delivery; Task analysis
• ISSN: 1545-5955; 1558-3783
• DOI: 10.1109/TASE.2023.3259905

This work presents a reinforcement learning-based switching control mechanism to autonomously move a ferromagnetic object (representing a milliscale robot) around obstacles within a constrained environment in the presence of disturbances. This mechanism can be used to navigate objects (e.g., capsule endoscopy, swarms of drug particles) through complex environments when active control is a necessity but where direct manipulation can be hazardous. The proposed control scheme consists of a switching control architecture implemented by two sub-controllers. The first sub-controller is designed to employ the robot's inverse kinematic solutions to do an environment search for the to-be-carried ferromagnetic particle while being robust to disturbances. The second sub-controller uses a customized rainbow algorithm to control a robotic arm, i.e., the UR5 robot, to carry a ferromagnetic particle to a desired position through a constrained environment. For the customized Rainbow algorithm, Quantile Huber loss from the Implicit Quantile Networks (IQN) algorithm and ResNet are employed. The proposed controller is first trained and tested in a real-time physics simulation engine (PyBullet). Afterward, the trained controller is transferred to a UR5 robot to remotely transport a ferromagnetic particle in a real-world scenario to demonstrate the applicability of the proposed approach. The experimental results on the UR5 robot show an average success rate of 98.86% over 30 episodes for randomly generated trajectories, demonstrating the viability of the proposed approach for real-life applications. In addition, two classical path finding approaches, Attractor Dynamics and the execution extended Rapidly-Exploring Random Trees (ERRT), are also investigated and compared to the RL-based method. The proposed RL-based algorithm is shown to achieve performance comparable to that of the tested classical path planners whilst being more robust to deploy in dynamical environments. Note to Practitioners -Deep reinforcement learning methods have been widely applied in computer games and simulations. However, employing these algorithms for practical, real-world applications such as robotics becomes challenging due to the difficulty of obtaining training samples. This paper predominantly focuses on bridging the gap between simulations and the real-world implementation of a reinforcement learning algorithm for a robotic application in the context of miniaturized drug delivery robots and robotic capsule endoscopes. This paper presents the derivation and experimental validation of a reinforcement learning-based algorithm for controlling a magnetically-actuated small-scale robot within a simplified model of the large intestine in the presence of disturbances. We demonstrate the possibility of training a high-fidelity reinforcement learning algorithm fully within a simulated environment before deploying it as-is in a real-world scenario by carrying out different experiments and simulations. Implementing the presented control framework complements a large body of this work, and the results offer a feasibility study of using reinforcement learning algorithms in practice.