Neural Network

Here let us study a supervised learning using a layered neural network. In a supervised learning, an conditional probability q(y|x) of an output y for a given input x is estimated. 

bayes041.mp4

Let an input x be in a 2-dimendional Euclidean space, and y in {0,1}. An unknown probability distribution q(y|x) is estimated by a three-layered neural network. In the movie, a network which is in a posterior distribution by MCMC is displayed. 


bayes042.mp4

An estimated posterior predictive distribution of a layered neural network is  displayed for each sample. The generalization error is far smaller than that by estimated using AIC. In a layered neural network, the essential dimension of a learning machine is far smaller than the dimension of the parameter. This is one of the main reason why deep learning has a better predictive performance than classical regular learning machines. 

A simple model selection procedure is introduced. FIve candidate models are compared according to the small generalization loss.

Generalization loss can be estimated by WAIC and LOOCV. Note that both of them are smaller than estimated by AIC. 

Theoretical performance of layered neural network (deep learning) has been studied in these 25 years.