[TBD]Role of positive definite matrix in machine learning

Introduction

Laymen explanation

Technical explanation

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Mathematical interpretation

Let

be the space of all vectors having real entries. A real matrix is said to be:

• 1. positive definite iff

• 1. for any non-zero ;

  2. positive semi-definite iff

• 1. for any ;

Properties of this matrix

• For positive definite matrix, All its eigenvalues are strictly positive real numbers. For positive sem-definite matrix, all its eigenvalues are non-negative real numbers (means that 0 is allowed)

• it is full-rank.

How to generate such matrix?

For any matrix, by using the property that a matrix multiplied by its transpose is positive semi-definte, you can generate a random positive semi-definite matix.

Similarly, Any covariance matrix is symmetric and positive semi-definite and its main diagonal contains variances (i.e., the covariance of each element with itself). So, generate the covariance matrix and be relaxed.

Use in machine learning

• • Positive semi definite matrix is used to decide if a ML loss function is convex or not(Refer here). Note that a convex loss function converges to global minima which is highly desirable property. (Refer here).

  • Convariance matrix in widely used PCA (for feature selection) is symmetric positive semi-definite matrix(SPD). PCA uses eigenvalue which can be complex number as well. this property ensure that Eigenvalues for are always real number.

  • Positive definite matrix is used in soft computing. This paper talks about this.

Reference

https://www.statlect.com/matrix-algebra/positive-definite-matrix

https://stackoverflow.com/questions/619335/a-simple-algorithm-for-generating-positive-semidefinite-matrices

https://en.wikipedia.org/wiki/Covariance_matrix

https://youtu.be/MLlk6h692BQ?t=2326

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6879&rep=rep1&type=pdf

https://sites.google.com/site/jbsakabffoi12449ujkn/home/machine-intelligence/role-of-convex-set-in-machine-learning

https://sites.google.com/site/jbsakabffoi12449ujkn/home/machine-intelligence/knowing-what-makes-ml-training-converge#TOC-Convergence-minima