Synthesis of Style-Specific Dance by Learning Body Gestures from Music

Abstract

Synthesis of dance from music is a very challenging task because the characteristics of swift and sequential movement of major parts of the body in a video are required to be preserved using a small set of audio and motion data. This paper proposes a novel architecture to learn dance of different styles including Ballet, Rumba, Cha-Cha, Tango and Waltz from musical videos. In particular, a deep learning architecture comprising well-known convolutional neural network and long short-term memory, and a novel probabilistic learning model, named, the mixture density networks are used to generate rhythmic movements of stick diagram from music. Then a suitable generative adversarial network is integrated to synthesize realistic dance from the rhythmic movements of the stick diagram. Experiments have been carried out on the YouTube and Motion Capture datasets to evaluate the performance of the proposed model. Results reveal that the proposed algorithm outperforms the existing ones both in terms of commonly-used performance indices

Keywords: Learning Gestures, Music to Dance, Synthesis of Dance, Image to image Translation