Cliprefiner: Enhancing Realism and Detail in Free-Hand Sketches Through Semantically-Aware Optimization

Abstract

Free-hand sketches are a fundamental tool for individuals to translate their internal perceptions of the world, bridging the gap between the abstract and the concrete. However, not everyone possesses the innate ability to convey their ideas through sketches, often resulting in sketches with certain limitations. Consequently, refining rough sketches into polished representations presents an intriguing challenge for both humans and machines. This study introduces CLIPRefiner, an innovative method that harnesses a pre-trained CLIP model to transform rough sketches from various categories into polished renditions. CLIPRefiner smoothens and optimizes two sets of Bézier curves, combining global and local optimization processes to enhance image semantics and enrich sketch details, all without the need for a dedicated sketch dataset for training. Notably, CLIPRefiner's local strokes initialization resampling significantly expedites the convergence process. Qualitative and quantitative experiments compellingly demonstrate CLIPRefiner's ability to transform rough sketches into high-quality, refined versions. The user study underscores the enhanced semantic perception of sketches refined by CLIPRefiner, while the ablation study provides valuable insights into the effectiveness of each module within CLIPRefiner. Additionally, CLIPRefiner has the potential to contribute a valuable rough-fine paired sketch dataset to the broader sketching community.

Keywords: Free-hand Sketches, CLIP, Bézier Curves, Rough-fine Sketch Transformation