What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?

Question Detail: 

What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?

As I recall your basic neural network is a 3 layers kinda thing, and I have had Deep Belief Systems described as being neural networks stacked on top of each other.

I've not til recently heard of a Deep Learning Systems, but i strongly suspect it is a synonym for Deep Belief System. Can anyone confirm this?

Asked By : Lyndon White

Best Answer from StackOverflow

Question Source : http://cs.stackexchange.com/questions/16545

Answered By : vzn

artificial neural networks models were generally restricted to only a few layers, say 3, for decades, for various reasons, including a math proof named Kolmogorovs thm that indicated they could theoretically approximate arbitrary functions with low error (but only with many neurons).

multilayer networks beyond that were not feasible/effective via prior learning techniques. a similar breakthrough came in 2006 by 3 different researchers Hinton, Le Cun, Bengio who were able to train much more layers. the prominent problem in this area was handwriting recognition.

the term "deep belief network" seems to originate with a 2006 paper by Hinton referring to Bayesian networks, which have close conceptual/theoretical connections/analogies with neural networks. "A fast learning algorith for deep belief nets"

see slide 17 in particular of this presentation deep neural networks

so the deep learning field is only ~½-decade old and is undergoing rapid research and development. Google, Facebook, Yahoo have all anounced deep-learning based initiatives and R&D is ongoing.