Test like you Train in Implicit Deep Learning

• Published in: arXiv.org
• Main Authors: Zaccharie Ramzi, Ablin, Pierre, Peyré, Gabriel, Moreau, Thomas
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 24.05.2023
• Subjects: Deep learning; Inference; Performance enhancement; Testing time; Training
• ISSN: 2331-8422

Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks (DEQs). In its general formulation, it relies on expressing some components of deep learning pipelines implicitly, typically via a root equation called the inner problem. In practice, the solution of the inner problem is approximated during training with an iterative procedure, usually with a fixed number of inner iterations. During inference, the inner problem needs to be solved with new data. A popular belief is that increasing the number of inner iterations compared to the one used during training yields better performance. In this paper, we question such an assumption and provide a detailed theoretical analysis in a simple setting. We demonstrate that overparametrization plays a key role: increasing the number of iterations at test time cannot improve performance for overparametrized networks. We validate our theory on an array of implicit deep-learning problems. DEQs, which are typically overparametrized, do not benefit from increasing the number of iterations at inference while meta-learning, which is typically not overparametrized, benefits from it.