Exploring Cross- and Within-Modality Representational Similarities in fMRI-Autoencoders Using PWCCA and RSA

Abstract

Examining fMRI signals modeled by deep neural networks has been challenging due to the lack of ground truth regarding underlying neural activity and the highly multidimensional nature of fMRI data. Drawing inspiration from advancements in fMRI-image reconstruction studies and recognizing the importance of model interpretability, this study revisits a foundational representational learning method, the autoencoder, for modeling fMRI signals. fMRI-autoencoders with different architectures are used to generate latent representations for both within- (between different autoencoder architectures) and cross-modality (between fMRI response and image stimuli) comparisons using two similarity measures: Projected Weighted Canonical Correlation Analysis (PWCCA) and Representational Similarity Analysis (RSA). Both similarity analyses reveal that across different fMRI-autoencoders, models with varying numbers of layers extract hierarchical information from fMRI signals differently, with deeper models generating representations with better separability between different object categories than shallow models. Cross-modality comparison reveals that latent fMRI representations constructed by any autoencoders effectively capture a substantial amount of information from images vectorized by any pretrained CNN-based models and are more similar to perceptual signals than semantic signals from the stimuli. Together, these analyses provide insights into representational learning between fMRI and image stimuli, demonstrating the utility of various similarity analyses in understanding multiscale, multidimensional, multimodal data.

Note:
Funding Information: No external funding was received

Conflict of Interests: The authors declare no competing interests.

Keywords: fMRI, autoencoder, representational learning, Deep learning, Multimodal