Fusing Visual and Mobility Data for City Sensing: A Case Study of Urban Village Recognition

Abstract

Using multimodal data fusion for city sensing can lead to more accurate, comprehensive and reliable results. Existing studies have used the strategy of fusion of remote sensing and social sensing to make the results least resource-intensive and most accurate. However, it is difficult to distinguish some similar urban functions using only remote sensing to represent visual information in spatial units. To fill the gap, this paper designs Sensing Blender, an end-to-end deep learning model that integrates remote sensing, street view imagery and social sensing to comprehensively characterize urban space. Specifically, this model combines physical environment features from satellite imagery and street view imagery with the dynamic mobility features from taxi trajectory data. In particular, a novel module in Sensing Blender is proposed to extract features from varying numbers of street view images. To validate the performance of the proposed model, a series of experiments of urban village recognition were conducted in Shenzhen, China, with a grid resolution of 500 meters. The results indicate that Sensing Blender achieved good performance with an overall accuracy (OA) of 92.0% and the Kappa of 0.720. Compared with unimodal models, our multimodal model improved the OA by 9.2% and the Kappa by 0.179. The proposed model provides an effective and efficient method for monitoring the distribution of urban villages, potentially supporting urban management, decision-making, and research on urban expansion and urban renewal.

Keywords: City Sensing, multimodal data fusion, Remote sensing, social sensing, deep learning, urban village recognition