An in Depth Introduction to Variational Bayes Note

Abstract

This note grew out of my attempt to understand variational inference (variational Bayes). Variational Bayesian approaches encompass a set of methodologies aimed at approximating inference problems that emerge within Bayesian inference and machine learning. These techniques have found extensive application across diverse fields of study. Within this note, we furnish a comprehensive guide to Variational Inference, with the specific intention of elucidating aspects often omitted in literature, thereby enhancing readers’ comprehension regarding the rationale behind the inclusion of particular equations in various papers and texts. This is accomplished by various examples ranging from simple to complex distributions. This note is inspired by Blei et al. (2003), Bishop (2006), Hoffman et al. (2010), Hoffman et al. (2013), Blei et al. (2017), Blei (2017), and Tran et al. (2021).

Keywords: Logistic regression, Massive data, Optimal subsampling, Newton’s method, Gradient descent, Stochastic gradient descent