Pose2Trajectory: Using transformers on body pose to predict tennis player’s trajectory

• Published in: Journal of visual communication and image representation Vol. 97; p. 103954
• Main Authors: AlShami, Ali, Boult, Terrance, Kalita, Jugal
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.12.2023
• Subjects: Body joints; Human pose estimation; Object detection; Body joints; Human pose estimation; Object detection
• ISSN: 1047-3203
• DOI: 10.1016/j.jvcir.2023.103954

Tracking the trajectory of tennis players can help camera operators in production. Predicting future movement enables cameras to automatically track and predict a player’s future trajectory without human intervention. It is also intellectually satisfying to predict future human movement in the context of complex physical tasks. Swift advancements in sports analytics and the wide availability of videos for tennis have inspired us to propose a novel method called Pose2Trajectory, which predicts a tennis player’s future trajectory as a sequence derived from their body joints’ data and ball position. Demonstrating impressive accuracy, our approach capitalizes on body joint information to provide a comprehensive understanding of the human body’s geometry and motion, thereby enhancing the prediction of the player’s trajectory. We use encoder–decoder Transformer architecture trained on the joints and trajectory information of the players with ball positions. The predicted sequence can provide information to help close-up cameras to keep tracking the tennis player, following centroid coordinates. We generate a high-quality dataset from multiple videos to assist tennis player movement prediction using object detection and human pose estimation methods. It contains bounding boxes and joint information for tennis players and ball positions in singles tennis games. Our method shows promising results in predicting the tennis player’s movement trajectory with different sequence prediction lengths using the joints and trajectory information with the ball position. •Pose2Trajectory, a novel model for predicting future trajectories of tennis players.•Encoder–decoder Transformer model trained on skelton data of tennis players.•Generated a high-quality skeleton-based dataset with the ball from multiple videos.•Model predicts tennis player trajectory for automated camera tracking.•Model reducing the need for human intervention in sports production.