[TBD]Role of cross-entropy loss in machine learning

Introduction

Laymen explanation

Cross-entropy(Is it same as maximum likelihood function? Refer https://machinelearningmastery.com/logistic-regression-with-maximum-likelihood-estimation/) is a commonly used loss function for classification tasks. Two examples that you may encounter include the logistic regression algorithm (a linear classification algorithm), and artificial neural networks that can be used for classification tasks.

If you love to know about its mathematics and uses, then this document is for you.

Technical explanation

Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions.

Mathematical explanation

Information(Entropy) h(x) can be calculated for an event x, given the probability of the event P(x) as follows:

h(x) = -log(P(x))

Binary cross entropy loss

Mathematics for ML classification problem

In classification tasks, we know the target probability distribution P for an input as the class label 0 or 1 interpreted as probabilities as “impossible” or “certain” respectively. These probabilities have no surprise at all, therefore they have no information content or zero entropy.

Cross-entropy model seeks to approximate the target probability distribution Q.

In the language of classification, these are the actual and the predicted probabilities, or y and yhat.

Expected Probability (y): The known probability of each class label for an example in the dataset (P).
Predicted Probability (yhat): The probability of each class label an example predicted by the model (Q).

We can estimate the cross-entropy for a single prediction using below formula

H(P, Q) = – sum x in X P(x) * log(Q(x))

The cross-entropy for a single example in a binary classification task can be stated by unrolling the sum operation as follows:

H(P, Q) = – (P(class0) * log(Q(class0)) + P(class1) * log(Q(class1)))

Uses

Multi-label classification

It can be applied using one-vs-all strategy

Check how?

Reference

https://images.app.goo.gl/EgGLsw4A6arCf8uK8

https://towardsdatascience.com/cross-entropy-for-classification-d98e7f974451

https://en.wikipedia.org/wiki/Entropy_(information_theory)

https://machinelearningmastery.com/cross-entropy-for-machine-learning/

Page updated

Google Sites

Report abuse