The Use of Autoencoders for Training Neural Networks with Mixed Categorical and Numerical Features

Abstract

We focus on modelling categorical features and improving predictive power of neural networks with mixed categorical and numerical features in supervised learning tasks. The goal of this paper is to challange the current dominant approach in the actuarial data science with a new architecture of a neural network and a new training algorithm. The key proposal is to use a joint embedding for all categorical features, instead of separate entity embeddings for categorical features, to learn a numerical representation of the categorical features which is fed, together with all other numerical features, into hidden layers of a neural network with a target response. In addition, we postulate that we should initalize the numerical representation of the categorical features and other parameters of the hidden layers of the neural network with parameters trained with denoising autoencoders in unsupervised learning tasks, instead of using random initialization of parameters. Denoising autoencoders for categorical data plays an important part in this research and they are investigated in more details in the paper. We illustrate our ideas with experiments on a real data set with claim numbers. We demonstrate that we can achieve a higher predictive power of the network
with our approach than with the current approach.

Keywords: Autoencoders, corruption of inputs, categorical and numerical features, embeddings, representation learning.