Β-Multivariational Autoencoder for Entangled Representation Learning in Video Frames

Abstract

In learning sequential decision-making where a set of actions are predicted from the states and previous reward, it is critical to choose actions from a proper distribution. Learning a known prior from data, on the other hand, becomes more challenging when the number of latent variables exceeds two and every two variables is related to each other via a covariance value. Moreover, many posterior estimation approaches experience posterior collapse when the data is large and diverse. In this paper, we propose the β-Multivariational Autoencoder (βMVAE) to learn a Multivariate Gaussian prior from video frames for use as part of a single object-tracking in form of a decision-making process. We present a novel formulation for object motion in videos with a set of dependent parameters to address a single object-tracking task. The true values of the motion parameters are obtained through data analysis on the training set. The parameters population is then assumed to have a Multivariate Gaussian distribution. The βMVAE is developed to learn this entangled prior p = N(µ,Σ) directly from frame patches where the output is the object masks of the frame patches. We devise a bottleneck to estimate the posterior’s parameters, i.e. µ',Σ'. Via a new reparameterization trick, we learn the likelihood p(xˆ|z) as the object mask of the input. Furthermore, we alter the neural network of βMVAE with the U-Net architecture and name the new network βMultivariational U-Net (βMVUnet). Our networks are trained from scratch via over 85k video frames for 24 (βMVUnet) and 78 (βMVAE) million steps. We show that βMVUnet enhances both posterior estimation and segmentation functioning over the test set. Our code and the trained networks are publicly released on Github.

Keywords: Representation learning, posterior estimation, variational inferences, Video object segmentation