A Framework for Assessing the Computational Reproducibility of Geo-Simulation Experiments

Abstract

Recent computational technologies have advanced geo-simulation experiments (GSEs), and the assessment of GSEs’ computational reproducibility has become increasingly critical. Some efforts aiming at some specific aspects, such as transparency and accessibility, lack a comprehensive view of assessment. This study proposes a framework focused on the assessment about reproducibility of computational processes in GESs. This framework focuses on the recognition of computations in GSEs, identifying the composition of computational workflow in GSEs at first. To complement the assessment procedure, this study proposes an assessment model designed to facilitate the assessment. At first, the conceptual model is structured around the dual-step assessment procedure: 1) examining the computational workflow across all computational processes, and 2) if expectations of assessment are not met, investigating clues of the unacceptability of expectations inside individual computational processes. Then, this assessment model with dimensions representing specific facets to be assessed and metrics defining detailed standards or guidelines for assessment. The entries in the dimensions and metrics are designed to reflect the unique characteristics of each dual-step assessment procedure. Finally, this study integrates the proposed framework into a prototype system and demonstrates the practicality of this framework through a detailed case study.

Keywords: Computational reproducibility, Geo-simulation experiment, Assessment framework