Sara: Controllable Makeup Transfer with Semantic-Guided Alignment and Region-Adaptive Normalization

Abstract

Makeup transfer is a process of transferring the makeup style from a reference image to the source image, while preserving the source image's identity. In addition to achieving fine-grained control over the makeup transfer, incorporating semantic alignment into the transformation is crucial, as the poses of the reference and source images are often inconsistent. We propose a novel Semantic-guided Alignment and Region-Adaptive normalization framework (SARA) to effectively transfer makeup styles under misaligned poses, offering flexible control to meet the demands of real-world applications, such as partial transfer, intensity adjustment, and makeup removal. Specifically, SARA comprises three modules: Firstly, we propose a semantic-guided alignment module to explicitly construct dense correspondence between the reference image and the target semantic map, employing unbalanced optimal transport matching to handle semantic region mismatches. Secondly, a region-adaptive normalization module is responsible for dynamically combining the warped style features with shape-independent style codes obtained by region-wise average pooling, mitigating feature loss during the alignment. Lastly, a makeup fusion module progressively fuses the identity features with the makeup styles to render the final output image. Furthermore, we combine optimal transport with histogram matching to generate the pseudo ground truth, which is used to facilitate the transfer in terms of both spatial alignment and color distribution. Experimental results show that our proposed SARA outperforms existing methods on two public datasets.

Keywords: Makeup transfer, Style transfer, Generative models