Non-Linear Dynamic Factor Analysis With a Transformer Network

Abstract

The article proposes a new machine learning Transformer algorithm, which can estimate dynamic factors from multivariate time series datasets. The performance is improved substantially on small datasets by using conventional factor models as prior information to guide the training. The Transformer consists of two Encoders. The first Encoder distills information from data into a factor estimate. The second Encoder has to rely on this factor estimate to minimize the prediction error. The Transformer learns through training to construct the factor estimate from data with minimal identifying assumptions. Monte Carlo experiment finds that the Transformer is more accurate than the Kalman filter, when the data deviates from linear-Gaussian.

Keywords: Transformer Network, Dynamic Factor, State Space