Physics Perception in Sloshing Scenes With Guaranteed Thermodynamic Consistency

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 2; pp. 2136 - 2150
• Main Authors: Moya, Beatriz, Badias, Alberto, Gonzalez, David, Chinesta, Francisco, Cueto, Elias
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Artificial neural networks; Augmented reality; Computer Science; Computer vision; Deep learning; Engineering Sciences; Fluids mechanics; Free surfaces; GENERIC; Heuristic algorithms; Image reconstruction; Liquids; Machine Learning; Manifolds; Mechanics; Modeling and Simulation; Neural networks; Perception; Physics perception; Real time; Real-time systems; Recurrent neural networks; sloshing; Thermodynamics; thermodynamics-aware deep learning; Vision systems; sloshing; thermodynamics-aware deep learning; Physics perception; GENERIC
• ISSN: 0162-8828; 1939-3539; 2160-9292; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2022.3160100

Physics perception very often faces the problem that only limited data or partial measurements on the scene are available. In this work, we propose a strategy to learn the full state of sloshing liquids from measurements of the free surface. Our approach is based on recurrent neural networks (RNN) that project the limited information available to a reduced-order manifold to not only reconstruct the unknown information but also be capable of performing fluid reasoning about future scenarios in real-time. To obtain physically consistent predictions, we train deep neural networks on the reduced-order manifold that, through the employ of inductive biases, ensure the fulfillment of the principles of thermodynamics. RNNs learn from history the required hidden information to correlate the limited information with the latent space where the simulation occurs. Finally, a decoder returns data to the high-dimensional manifold, to provide the user with insightful information in the form of augmented reality. This algorithm is connected to a computer vision system to test the performance of the proposed methodology with real information, resulting in a system capable of understanding and predicting future states of the observed fluid in real-time.