Cross-entropy(Is it same as maximum likelihood function? Refer https://machinelearningmastery.com/logistic-regression-with-maximum-likelihood-estimation/) is a commonly used loss function for classification tasks. Two examples that you may encounter include the logistic regression algorithm (a linear classification algorithm), and artificial neural networks that can be used for classification tasks.
If you love to know about its mathematics and uses, then this document is for you.
Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions.
Information(Entropy) h(x) can be calculated for an event x, given the probability of the event P(x) as follows:
h(x) = -log(P(x))
Binary cross entropy loss
In classification tasks, we know the target probability distribution P for an input as the class label 0 or 1 interpreted as probabilities as “impossible” or “certain” respectively. These probabilities have no surprise at all, therefore they have no information content or zero entropy.
Cross-entropy model seeks to approximate the target probability distribution Q.
In the language of classification, these are the actual and the predicted probabilities, or y and yhat.
Expected Probability (y): The known probability of each class label for an example in the dataset (P).
Predicted Probability (yhat): The probability of each class label an example predicted by the model (Q).
We can estimate the cross-entropy for a single prediction using below formula
H(P, Q) = – sum x in X P(x) * log(Q(x))
The cross-entropy for a single example in a binary classification task can be stated by unrolling the sum operation as follows:
H(P, Q) = – (P(class0) * log(Q(class0)) + P(class1) * log(Q(class1)))
Multi-label classification
It can be applied using one-vs-all strategy
Check how?
Reference
https://images.app.goo.gl/EgGLsw4A6arCf8uK8
https://towardsdatascience.com/cross-entropy-for-classification-d98e7f974451
https://en.wikipedia.org/wiki/Entropy_(information_theory)
https://machinelearningmastery.com/cross-entropy-for-machine-learning/