Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

Abstract

This paper presents a Geometric-Photometric Joint Alignment(GPJA) method, which aligns discrete human expressions at pixel-level by combining geometric and photometric information. Common practices for registering human heads typically involve aligning landmarks with facial template meshes using geometry processing approaches, but often overlook dense pixel-level photometric consistency. This oversight leads to inconsistent texture parameterization across different expressions, hindering the creation of topologically consistent head meshes widely used in movies and games. GPJA overcomes this limitation by leveraging differentiable rendering to align vertices with target expressions, achieving joint alignment in both geometry and photometric appearances automatically, without requiring semantic annotation or pre-aligned meshes for training. It features a holistic rendering alignment strategy and a multiscale regularized optimization for robust and fast convergence on large-step deformation. The method utilizes derivatives at vertex positions for supervision and employs a gradient-based algorithm which guarantees smoothness and avoids topological defects during the geometry evolution. Experimental results demonstrate faithful alignment under various expressions, surpassing the conventional non-rigid ICP-based methods and the state-of-the-art deep learning based method. In practical, our semantic annotation-free method enhances the efficiency of obtaining topology-consistent face models from multi-view stereo facial scanning.

Keywords: Geometry registration, Facial performance capture, Face modeling