Dual-Task Cascaded Network for Spatial-Temporal-Spectral Image Fusion in Remote Sensing

Abstract

Due to hardware constraints, no satellite sensor can supply images with all high spatial, high temporal and high spectral resolutions nowadays. Spatialtemporal-spectral fusion (STSF) is supposed as an economical and feasible approach to tackle this contradiction. However, most works still have drawbacks: on the one hand, the studies deployed on MODIS and Landsat data cannot be transferred to spaceborne hyperspectral (HS) data with lower temporal resolution; on the other hand, the rigid time correlation function exhibits weakness orienting to non-linear land cover changes. In this paper, we propose a novel dual-task cascaded network that incorporates spatialspectral fusion and temporal-spectral fusion into a unified end-to-end structure. Wherein the former develops a scale alternating projection module with coupled space enhancement unit and error correction unit to enhance realistic spatial textures to obtain latent high spatial resolution (HR)-HS image. The latter designs a spectral fine tuning network to revise spectral curve value and then inserts a difference image for temporal mutation region awareness. Extensive experiments were implemented on Ziyuan(ZY)-1 02D HS data and Sentinel-2 multispectral (MS) data. Both qualitative and quantitative results demonstrated that the proposed method raised spectral quality on variations and achieved the superior performance comparing to previous algorithms.

Keywords: Spatial-temporal-spectral fusion, Hyperspectral image, Multispectral image, Temporal change, Remote Sensing