Robotic Visual GUI Testing for Truly Non-Intrusive Test Automation of Touch Screen Applications

• Published in: IEEE transactions on software engineering Vol. 51; no. 4; pp. 1205 - 1231
• Main Authors: Qian, Ju, Lv, Guizhou, Jin, Yiming, Shang, Zhengyu, Yan, Shuoyan, Wang, Yan, Chen, Lin
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2025; IEEE Computer Society
• Subjects: Automation; Computer vision; Consoles; Engines; Games; Graphical user interfaces; GUI testing; Interactive computer systems; non-intrusive; Robot arms; Robot kinematics; robotic testing; Robots; Scripts; Smartphones; Software; Switches; Test engines; Testing; Touch screens; Touch sensitive screens; User interface; Visualization
• ISSN: 0098-5589; 1939-3520
• DOI: 10.1109/TSE.2025.3544441

Test automation intrusive to the devices under test is difficult to apply on closed or uncommon touch screen systems, e.g., a Switch game console or a digital instrument running a self-defined operating system. There is a lack of non-intrusive test automation techniques for situations where intrusive testing is impossible or not easy to apply. This paper presents RoScript, a novel robotic visual GUI testing system for truly non-intrusive test automation of touch screen applications. RoScript expresses GUI actions in visual test scripts and executes them via a physical robot. A key innovation of RoScript is a test engine armed with environment calibration techniques to achieve automated test execution without manually setting any environment parameter or adjusting the robot arms for a new subject under test. Additionally, two complementary computer vision-based methods are also introduced to record test scripts from videos of human actions on a touch screen. The RoScript test automation does not rely on the internal system of a device under test, making it truly non-intrusive and suitable for touch screen applications running on almost any platform. We evaluated RoScript on a diverse range of devices--including three Android/iOS phones, a Windows tablet, a Linux-based Raspberry Pi, a GoPro camera, and a Switch game console--across over 1100 GUI actions in 160 test scenarios. The results demonstrate RoScript's high accuracy in test execution: 94% for executing test scripts and 97% for replicating GUI actions. Furthermore, RoScript accurately recorded about 85% of human touch screen actions into test code. These results highlight RoScript's potential as a truly non-intrusive, cross-platform solution for GUI test automation.