CasFormer: Cascaded transformers for fusion-aware computational hyperspectral imaging

• Published in: Information fusion Vol. 108; p. 102408
• Main Authors: Li, Chenyu, Zhang, Bing, Hong, Danfeng, Zhou, Jun, Vivone, Gemine, Li, Shutao, Chanussot, Jocelyn
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.08.2024
• Subjects: Artificial intelligence; Cascade attention; CASSI; Computational imaging; Fusion; Hyperspectral; RGB; Spatial; Spectral; Transformer; Transformer; CASSI; Spatial; Hyperspectral; Spectral; Cascade attention; RGB; Artificial intelligence; Computational imaging; Fusion
• ISSN: 1566-2535; 1872-6305; 1872-6305
• DOI: 10.1016/j.inffus.2024.102408

Computational hyperspectral imaging (CHI) is a cutting-edge technique, which plays a pivotal role in breaking through the quality bottleneck of hyperspectral images (HSI). Among the techniques employed in this domain, the coded aperture snapshot spectral imaging (CASSI) system holds widespread recognition. Nevertheless, the imaging capability of CASSI remains limited due to the hardware conditions and the fragility of outcomes associated with the ill-posed blind reconstruction process. To this end, we propose a novel cascaded transformer architecture, termed CasFormer, specifically crafted for fusion-aware CHI by means of a dual-imaging mechanism. CasFormer facilitates the effective enhancement of hyperspectral imaging quality by fusing RGB images, with a focus on spatial and spectral domains. As the name suggests, CasFormer is primarily composed of a series of cascade-attention blocks, enabling the fusion of high-spatial-resolution RGB images through spatial coherence alignment and the recovery of spectrally sequential information more compactly and accurately. Furthermore, CasFormer incorporates physical constraints through a decoupling-based loss function, ensuring spatial consistency and spectral fidelity in the fusion-aware CHI process. Extensive experiments conducted across multiple datasets demonstrate the superiority of CasFormer in achieving high-quality imaging results compared to SOTA CHI algorithms. Our code and benchmark datasets will be openly accessible at https://github.com/danfenghong/Information_Fusion_CasFormer. •Propose a cascaded transformer for fusion-aware hyperspectral computational imaging.•Fuse RGB images and compressive CASSI measurements for hyperspectral imaging.•Design spectral-aware self-attention and spatial-fused cross-attention.•Show effectiveness and superiority on three different datasets.