Theory Story
DNN recently emerges from a long history of neural networks with two empty periods. Since its beginning, more and more sophisticated concepts and related architectures were developed for neural networks and after for deep neural networks. Full surveys were provided by Schmidhuber in 2015, Yi et al. in 2016, Liu et al. in 2017, and Gu et al. in 2018. In addition, a full description of the different DNN concepts are available at the Neural Network Zoo website. Here we briefly summarize the main steps of the DNN’s story.
Beginning: DNN begins in 1943 with the threshold logic unit (TLU) In further works, Rosenblatt designed the first 5 perceptron in 1957 whilst Widrow developed the Adaptive Linear Neuron (ADALINE) in 1962. This first generation of neural networks are fundamentally limited in what they can learn to do.
First Empty Period: During the 1970s, research focused more on XOR problem.
Second Active Period: The next period concerns the emergence of more advanced neural networks like multilayer back-propagation neural networks, Convolutional Neural Networks (CNNs), and Long Short-Term Memory (LSTMs) for Recurrent Neural Networks (RNNs) . This second generation of neural networks mostly used back-propagation of the error signal to get derivatives for learning.
Second Empty Period: After 1995 until 2006 , research focused more upport Vector Machine (SVM) which is a very clever type of perceptron developed by Vapnik et al.. Thus, many researchers abandoned neural networks research with multiple adaptive hidden layers because SVM worked better with less computational time requirements and training.
Third Active Period: With the progress of GPU and the storage of Big Data, DNN regains attention and developments with new deep learning concepts such as a) Deep Belief Networks in 2006 and b) Generative Adversarial Networks (GANs) in 2014.
Liu et al. classified the deep neural network architectures in the following categories: restricted Boltzmann machines (RBMs), deep belief networks(DBNs), autoencoders(AEs) networkanddeep ConvolutionalNeuralNetwork (CNNs). In addition, deep probabilistic neural networks, deep fuzzy neural networks and Generative Adversarial Networks (GANs) can also be considered as other categories.
Architecture Story
Applications of these deep learning architecture are mainly in speech recognition, computer vision and pattern recognition. In this context, DeepNets architectures for specific applications have emerged such as the following well-known architecture:
AlexNet developedby Krizhevsky et al. [104] for image classification in 2012.
VGG-Net designed by Simonyan and Zisserman for large-scale image recognition in 2015.
U-Net developed by Ronneberger et al. for biomedical image segmentation in 2015.
GoogLeNet with inception neural network introduced by Szegedy et al. for computer vision in 2015.
Microsoft Residual Network (ResNet) designed by He et al. for image recognition in 2016.
Thus, all these architectures were designed for a target application like speech recognition, computer vision and pattern recognition which its specific features giving very impressive performance in comparison on the previous state-of-art methods based model-based methods.