Optimizing Multi-Task Network with Learned Prototypes for Weakly Supervised Semantic Segmentation

Abstract

Weakly supervised semantic segmentation (WSSS) presents a challenging task wherein semantic objects are extracted solely through the utilization of image-level labels as supervision. One common category of state-of-the-art (SOTA) solutions depends on the generation of pseudo pixel-level annotations via localization maps. Nevertheless, in the majority of such solutions, the quality of pseudo annotations may not effectively fulfill the requirements of semantic segmentation owing to the incomplete nature of the localization maps. In order to generate denser localization maps for WSSS, we propose the use of a prototype learning guided multi-task network. Initially, the prototypical feature vectors are employed to depict the similarities between images. Specifically, the shared information among different training images is thoroughly exploited to concomitantly learn the prototypes for both foreground categories and background. This approach facilitates the localization of more reliable background pixels and foreground regions by evaluating the similarities between the representative prototypes and the extracted features of pixels. Additionally, the learned prototypes can be incorporated into the multi-task network to enhance the efficiency of parameter optimization by adaptively rectifying errors in pixel-level supervision. Therefore, the optimization of the multi-task network for object localization and the production of high-quality proxy annotations can be achieved by means of clean image-level labels and refined pixel-level supervision working in conjunction. Extensive experiments conducted on two datasets, namely, PASCAL VOC 2012 and COCO 2014, have substantiated the fact that the prototype learning guided multi-task network being proposed outperforms the current SOTA methods in terms of segmentation performance.

Keywords: Weakly Supervised Semantic Segmentation, Multi-task Network, Prototype Learning