auto encoder

An autoencoder is a type of neural network for encoding data with high dimentionality. It learns efficient encoding of data in an unsupervised manner, and it's usually used for dimention reduction and removing noise.

The strucutre is simple. It feeds input data into hidden layers and then a vector (code) with a given number of dimensions (usually much smaller than the original input). This part is called encoder. It represents the original data with a smaller vector/code.

Then the vector is further fed into another set of hidden layers aiming to reconstruct the original input data as output. This second part is called decoder.

So the neural work feeds in data, reduces its dimension and restores the original data. No labelling is needed. Only the original data is used.

input --> hidden layer --> hidden layer ... --> vector(code) --> hidden layer --> hidden layer --> output

By doing this dimension reduction and data reconstruction, it hopes to learn about some essential features / any useful information / patterns from the data, and use those features to encode the data.

Not surpising it may just learn an identity function, i.e. simply flow input through as output in someway.

So different regularisation methods are developed to prevent that.

a few examples are:

Sparse autoencoder

Denosing autoencoder

Contrctive autoencoder

Note, when the number of hidden layers is 1, the auto encoder strongly correclates to PCA principal component analysis.