Improved Architectures and Training Algorithms for Deep Operator Networks

• Published in: Journal of scientific computing Vol. 92; no. 2; p. 35
• Main Authors: Wang, Sifan, Wang, Hanwen, Perdikaris, Paris
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2022; Springer Nature B.V; Springer
• Subjects: Algorithms; Back propagation; Banach spaces; Benchmarks; Bias; Boundary conditions; Computational Mathematics and Numerical Analysis; Mathematical and Computational Engineering; Mathematical and Computational Physics; Mathematics; Mathematics and Statistics; Neural networks; Operators (mathematics); Partial differential equations; Physics; Theoretical; Deep learning; Physics-informed machine learning; Computational physics; Partial differential equations
• ISSN: 0885-7474; 1573-7691; 1573-7691
• DOI: 10.1007/s10915-022-01881-0

Operator learning techniques have recently emerged as a powerful tool for learning maps between infinite-dimensional Banach spaces. Trained under appropriate constraints, they can also be effective in learning the solution operator of partial differential equations (PDEs) in an entirely self-supervised manner. In this work we analyze the training dynamics of deep operator networks (DeepONets) through the lens of Neural Tangent Kernel theory, and reveal a bias that favors the approximation of functions with larger magnitudes. To correct this bias we propose to adaptively re-weight the importance of each training example, and demonstrate how this procedure can effectively balance the magnitude of back-propagated gradients during training via gradient descent. We also propose a novel network architecture that is more resilient to vanishing gradient pathologies. Taken together, our developments provide new insights into the training of DeepONets and consistently improve their predictive accuracy by a factor of 10-50x, demonstrated in the challenging setting of learning PDE solution operators in the absence of paired input-output observations. All code and data accompanying this manuscript will be made publicly available at https://github.com/PredictiveIntelligenceLab/ImprovedDeepONets .