Role of Bayes decision rule in machine learning

Introduction

Laymen explanation

In finance, Bayes' theorem is used to rate the risk of lending money to potential borrowers. If you are interested to know about its usage in machine learning, then this document helps.

Technical explanation

Bayesian decision theory refers to a decision theory which is informed by Bayesian probability. It is a statistical system that tries to quantify the tradeoff between various decisions, making use of probabilities and costs.

Mathematical interpretation

Mathematically, it solves problem of designing classifiers to recommend decisions that minimize some total expected ”risk”

– The simplest risk is the classification error (i.e., costs are equal).

– Typically, the risk includes the cost associated with different decisions.

Below diagram shows the risk value in probabilities terminology (Refer: Integral section pointing arrow)

Properties of Bayes Decision rule

It provides optimal classification rule. With mathematical proof, it tells that no classifier in the world can have lower mis-classification error probability than Bayes one (bayesian probability)

Role in machine learning

• Bayesian probability gives the benchmark probability for any classification based model. In other words, no classification model can be designed which can perform better than bayesian probability.

• It can be used for verifying the performance of any Machine learning model. Closer the value to the bayesian probability, better the model.

• Bayesian Optimization is often used in applied machine learning to tune the hyperparameters of a given well-performing model on a validation dataset.

  • Reason behind good performance of popular multi-layer perceptron(deep neural network) based classifier has been mathematically supported by comparing its performance with Bayesian probability

  • It is used to find optimal k value for K-nn classifier

Why we need non Bayes classifier then?

Although Bayes is best, but implementing it for real world problems are complicated. For example, integration calculation which is needed for Bayes model can't be done in case class shape is complicated.

Reference

https://youtu.be/0SNSVjeTXao?t=1034

https://stephens999.github.io/fiveMinuteStats/decision_theory_bayes_rule.html

https://www.cc.gatech.edu/~hic/CS7616/pdf/lecture2.pdf

https://images.app.goo.gl/FRBmN2qk93gzXsDE7

https://www.investopedia.com/terms/b/bayes-theorem.asp

https://machinelearningmastery.com/bayes-theorem-for-machine-learning/

https://link.springer.com/article/10.1007/s12530-020-09345-2

https://youtu.be/0SNSVjeTXao?t=3410

https://youtu.be/0SNSVjeTXao?t=3759

https://youtu.be/DlQli0OCkf8?t=2040