Deep Belief Networks

Deep Belief Networks (DBNs) are a type of artificial neural network that are composed of multiple layers of hidden units, specifically, they are a class of deep neural network which comprises several layers of latent variables or hidden units. DBNs have two main types of layers: visible layers and hidden layers. The visible layers are those that receive inputs from the outside world, while the hidden layers are those that process inputs received from either visible layers or other hidden layers.

How they work

DBNs are made up of several Restricted Boltzmann Machines (RBM) where each RBM layer communicates with both the previous and next layers, but the individual layers are not connected amongst themselves. They're stacked up in such a way that the output of each layer is the input of the next. The connections are bidirectional.

The learning process in DBNs consists of two main phases: an unsupervised pre-training phase and a supervised fine-tuning phase.

In the pre-training phase, a DBN learns to reconstruct its inputs in an unsupervised manner layer by layer. Each layer is trained as an RBM which tries to learn the structure in the input received from the previous layer and then gives out the output to the next layer.

After pre-training, the outputs of the final hidden layer are used as inputs for a standard supervised learning algorithm like logistic regression in the fine-tuning phase.

So, initially, the DBN attempts to understand the structure in the input by trying to model the input distribution, then it fine-tunes the parameters using a labelled dataset to perform tasks such as classification or regression.

The two-phase learning system makes DBNs quite effective at handling unlabeled data, as they can make useful abstractions from the data in the pre-training phase and then fine-tune using a smaller set of labeled data.