Perspective-Adaptive Convolutions for Scene Parsing

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 42; no. 4; pp. 909 - 924
• Main Authors: Zhang, Rui, Tang, Sheng, Zhang, Yongdong, Li, Jintao, Yan, Shuicheng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.04.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; context adaptive biases; Convolutional neural networks; Feature maps; perspective-adaptive convolutions; Proposals; Scene parsing; Shape; Spatial dependencies; Strain; Task analysis; Training; Visibility
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2018.2890637

Many existing scene parsing methods adopt Convolutional Neural Networks with receptive fields of fixed sizes and shapes, which frequently results in inconsistent predictions of large objects and invisibility of small objects. To tackle this issue, we propose perspective-adaptive convolutions to acquire receptive fields of flexible sizes and shapes during scene parsing. Through adding a new perspective regression layer, we can dynamically infer the position-adaptive perspective coefficient vectors utilized to reshape the convolutional patches. Consequently, the receptive fields can be adjusted automatically according to the various sizes and perspective deformations of the objects in scene images. Our proposed convolutions are differentiable to learn the convolutional parameters and perspective coefficients in an end-to-end way without any extra training supervision of object sizes. Furthermore, considering that the standard convolutions lack contextual information and spatial dependencies, we propose a context adaptive bias to capture both local and global contextual information through average pooling on the local feature patches and global feature maps, followed by flexible attentive summing to the convolutional results. The attentive weights are position-adaptive and context-aware, and can be learned through adding an additional context regression layer. Experiments on Cityscapes and ADE20K datasets well demonstrate the effectiveness of the proposed methods.