Deep Boltzmann machines

We present a new learning algorithm for Boltz-mann machines that contain many layers of hid-den variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and data-independent expectations are approximated us-ing persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden lay-ers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training ” phase that allows variational in-ference to be initialized with a single bottom-up pass. We…