Basic Statistics for Deep Learning

Median: Median is the number that separates the higher half of a data sample, a population, or a probability distribution, from the lower half.

Let's arrange the dataset in the ascending order i.e. 1,2,2,2,3,14

There as two middle values in the 3rd and 4th position from the left which are separators. The real value of the separator in this even number of  dataset is the mean of these two middle values at n/2th and (n/2 + 1)th position i.e.

2+2/2 = 2

For odd data number of dataset it will be n/2 + 1. The median seems to be more robust to outliers than mean.

Mode: Mode is the value that appears most in the dataset i.e. the highest frequency element. In the given data the frequency for 2 is the highest i.e. 3 and hence 2 is the mode. Note that for a normal distribution, mean, median, and mode are the same.

Range: Range in a data set is the difference between the largest and the smallest value. In the given data set, it is:

14 - 1 = 13

Variance: Variance is the measure of how far the set of number in a dataset are spread out. The small variance implies that the data points tend to be closer to mean and vice versa. In the given data set, deviations of numbers from mean are:

Deviation from mean of X = {1-4 = -3, 2 - 4=-2, 3-4=-1, 14-4=10}

. Then,

Odd Ratio (OR) is the ration of two Odds. Log of Odds is the transformation from probabilities to log of odds to get around with restricted range of [0  1] to [-inf inf]. It is also easy to interpret and understand. It is represented by logit function.

Titanic Example (http://pages.uoregon.edu/aarong/teaching/G4075_Outline/node15.html)

What was the overall odds (both male and female) of survival of Titanic if 38% passengers survived?

O (survival) = 0.38/(1 - 0.38) = 0.61

So, for every single death on the Titanic, there was an average 0.61 survivors.

Now, let’s look at the odd ratio of survival of men and women in the Titanic where 19% of men and 73% of women survived.

O (women) = 2.67 & O (men) = 0.24

OR = O(women)/O(men) = 11.125

It indicates that women were 11 times more likely as men to survive the Titanic.

Probability Distribution

It is the list of all values that the random variable can assume with their corresponding probabilities. It is the set of all possible outcomes of the random phenomenon being observed. 

For rolling a dice, each outcome has the probability of 1/6 and the list of values are {1*1/6, 2*1/6, ... 6*1/6).

E[X] = 3.5, Var(X)=2.9 & SD=1.7078

Normal (Gaussian) Distribution or Bell Curve

It is the continuous probability distribution that follows the Central Theorem. There is a symmetry between the left and right i.e. data is distributed equally around the central value called mean. The value of the normal distribution diminishes and is practically zero when the value X lies more than a few standard deviations away from the mean. The mean, median and more are all equal in normal distribution as showed in Fig. 1 

 Fig. 2: Normal Distribution demonstrating 68-95-99.7 Rule [1]

Central Theorem: Given a certain condition, the arithmetic mean of a sufficiently large number of iterations of independent random variables, each with a well defined expected value and variance, will be approximately normally distributed regardless of the underlying distribution.

Example:

Let's roll n 6-sided fair dices where n is very large, then the distribution of the sum of the rolled numbers i.e. sum of outcome of dices, as a result of random events, will be well approximated by a normal distribution which follows the central theorem. Let's create probability distribution of sum of outcomes.

Let's consider two dices, A and B i.e. n=2. Then, it will have the sample space of 36 outcomes as showed in table 1 below.

Table 1: Sample space of outcomes by rolling two dices [1]

Total sum of all possible outcomes is 252 which gives the mean of 252/36 = 7.

Let's calculate the probability of each outcome out of all possible outcomes.

Data 2 and 12 has the lowest probability of 1/36 and data 7 has the highest probability of 6/36 = 1/6. Similarly, the probability distribution with larger value of n can be calculated which results in probability distribution as in Fig. 3.

Fig. 3: Probability density of sum of outcomes from rolling n number of dices [1]

Bernoulli Distribution

It is a probability distribution of a random variable which takes the value 1 with a success probability "p" and 0 with the failure probability "q".

if X is a random variable, then Pr(X=1) = p & Pr(X=0) = 1 - p = q. In the example above, rolling two dices, the success probability for random variable 7 is 1/6. Then the failure probability will be 1 - 1/6 = 5/6.

In terms of tuples,

f(k;p) = p if k = 1 & 1 - p if k = 0

This can be re-written as:

In Normal distribution, most values (in fact about 95% of them) lie between μ − 2σ and μ + 2σ . As the distribution of sample means has mean "m" and standard deviation equal to the "SE", there is a 95% chance that the sample mean is between μ ± 2SE, which amounts to saying that there is a 95% chance that μ lies between m ± 2SE and this interval is called the Confidence Interval (CI).

CI = (4 - 2*2.0167, 4 + 2*2.0167) = (0.0334,8.03334)

Regressions

Regression is a statistical process for estimating a relationship among variables - a dependent variable and one or more independent variable(s) or predictors. The dependent variable represent the outcome or variation whereas the independent variable represent input or causes which is the potential reason for variation in output.

In a function, Y = f(X) = X, Y is the dependent variable which depends on X.

Regression analysis helps to understand how the typical value of dependent variable changes when any one of the independent variable is varied holding other variables fixed.

Linear Regression

Here the predictors depend linearly on the unknown independent parameters.

Case study [4]: 

Based on the math aptitude test score, the department of statistics is selecting the 5 students based on three questions:

• 1. What linear regression best predicts statistics performance based on scores?

  2. If a student had scored 80 on the aptitude test, what grade would we expect him/her to make in statistics?

  3. How well the regression equation fits the data?

So, the value of b can be calculated as:

Similarly, the value of m is calculated as:

Plugging in the values, we get

m = 0.644 & b = 26.768

It is the best practice to represent the problem with a cost function or loss function for the average sum of squared errors as showed.

Level of Statistical significance: It is a threshold level below which the null hypothesis will be rejected. Typically, alpha = 0.05 indicates a 5% risk of inferring that if there was a 5% or less chance, one would reject the Null hypothesis and accept the alternative hypothesis.Given that the Null Hypothesis is true, how likely would it be to see the decrease in mean grades in Statistics?  In the example p-value  or the significance level is 0.1945

P-Value: P-Value is a statistical measure that determines whether or not the hypothesis are correct or are of significance. P-values are the probability of obtaining an effect at least as extreme as the one in your sample data, assuming the truth of the null hypothesis.

Example: A company prints baseball cards and it claims that 30% of the cards are rookies, 60% veterans and 10% all star. Let’s calculates its p-value using 0.05 (alpha) level of significance i.e. below this value, the hypothesis or claim will be rejected.

Suppose a random sample of 100 cards has 50 rookies, 45 veterans and 5 all stars

There are 3 random variables i..e k = 3 and df = k - 1 = 3 - 1 = 2

The Observed counts are:

Orookie= 50, Oveteran = 45, and Oallstar = 5

Based on the claims, out of 100 card, the expected baseball cards are:

Erookie = 30, Eveteran=60 and Eallstar = 10

Then, the probability becomes:

Conditional Probability: It is the measure of the probability of event given that other event has occurred. 

In table 1, the probability of getting 2 on dice A i.e

Pr(A=2) = 6/36 = 1/6

Similarly, 

Pr(A+B<=5) = 10/36

So, the probability of getting 2 on dice A given the space of outcome 10 where the outcome is smaller than equal to 5 and is revealed, is give by:

Pr(A=2 | A+B <=5) = 3/10

Case Study [5]: 

Consider a outcome variable "hon" indicating whether the student is in the honor class or not and the predictor variables gender, math and reading. The scatter plot, Honor Vs Math, shows the categorical outcomes of 0 (No) or 1 (Yes).

 Case I: with No predictor variable; only the intercept i.e. Y = β0

Check the Frequency table for honors.

hon freq 

 0  151 

 1  49

The overall probability of being in the honors class is 49/200 = 0.245. So, the odds of being in honor class = .245/(1 - .245) = .3245

logit(.3245) = ln(.3245) = -1.2547. In other words, the intercept (β0 = -1.2547) from the model with no predictor variables is the estimated log odds of being in honors class for the whole population of interest.

Case II: with a single dichotomous predictor variable i.e. Y = β0 + β1*female

Check the Frequency table for female predictor (0=>male; 1=>female)

 hon   0   1

  0   74  77

  1   17  32

Calculate the odds of being in the Honors Class:

Pr (male) = 17/91

O (male) = (17/91)/(1 - 17/91) = (17/91)/(74/91) = 17/74 = 0.23

O (female) = 32/77 = 0.42

log(O(male)) = ln (0.23) = -1.4697 (B0 ~ -1.470852 (from model)

log(O(female)) = ln (0.42) = -0.8675

The ratio of the odds for female to the odds for male = 0.42/0.23 = 1.809 i.e. the odds for female are about 81% higher than the odds for males.

B1 = ln (1.809) = 0.59277 ~ .59278 (from model)

The intercept of -1.471 is the log odds for males since male is the reference group (female = 0). The coefficient for females is the log of odds ratio between the female group and male group: log(1.809) = .593

Case III: with a single continuous Predictor variable i.e. Y = β0 + β1*math

The plot with actual logistic regression is showed below:

The intercept in this model (β0 = -9.79394) corresponds to the log odds of being in an honors class when math is at the hypothetical value of zero. In other words, the odds of being in an honors class when the math score is zero is exp(-9.793942) = .00005579 which is very low.

Substituting the value of B0 and B1, and plugging in math scores 54 and 55, in the logit equation, we can find that for a one-unit increase in math score, we expect to see about 17% increase in the odds of being in an honors class. This 17% of increase does not depend on the value that math is held at. 

Case IV: with all predictors 

By holding math and reading at a fixed value, the odds of getting into an honors class for females (female = 1) over the odds of getting into an honors class for males (female = 0) is exp(.979948) = 2.66 i.e. the odds for females are 166% higher than the odds for males. The coefficient for math says that, holding female and reading at a fixed value, we will see 13% increase in the odds of getting into an honors class for a one-unit increase in math score since exp(.1229589) = 1.13. 

Maximum Likelihood estimation (MLE)

MLE is a method of estimating the parameters of a statistical model given only the sample data (observed data) from the overall population. It finds the value of the parameters which makes the observed data most likely to have occurred. It is useful in cases where it is infeasible or expensive to measure each individual parameter.

Example [6]: 30%,45%, and 50% of birds from location A, B, and C are infected with West Nile virus. There is a sample of 100 birds either from location A, B or C with 40 cases of Nile virus. Given this observed data which value of p (i.e. probability parameter) makes the observed data most likely to have occurred? 

From the R  calculations for Binomial distribution given below (see Section: using Software), at p = 0.45, likelihood is maximized for the observed data. So, the sample has higher probability to contain birds that belong to location B. 

> dbinom(40,100,.45)

[1] 0.0488029

> dbinom(40,100,.5)

[1] 0.01084387

> dbinom(40,100,.3)

[1] 0.008490169

The likelihood function in terms of p can be written as

L(p) = (n k) pk (1-p)n-k

And, it is maximized when

d(L(p))/dp = 0

which gives,

 p = k/n = 40/100 = 0.4

So, the maximum likelihood estimate for p is p'=0.4 where the likelihood is maximized,

> dbinom(40,100,.4)

[1] 0.08121914

Using MLE we can help determine the value of β0 and β1 of the logistic regression that make the observed data most likely to have occurred. In the logistic regression example, the probability of honor p(xi) given math grade equals to xi is given by:

p(xi) = e^( β0 +  β1*xi )/(1 - e^( β0 +  β1*xi)

Since p is the function in x, and y is the observed dataset which is either 0 or 1, the likelihood for each input data point in terms of Bernoulli distribution then becomes [7],

That’s a transcendental equation, and there is no closed-form solution for this equation.We can however approximately solve it numerically. Using Newton-Ralphson method, it becomes:

βn+1 = βn - f’(βn)/f’’(βn)

For, details on Newton Raphson method, please find the Reference Guide.

Using Software

Request a compute node

srun --pty bash

Load the appropriate module

module load R

Calculate the statistical measures:

The psych library is available in R/3.2.0. You may need to install psych library following the Software Installation Guide

module load R/3.2.0

R

> library(psych)

mydata = c(2, 1, 3, 2, 2, 14)

> describe(mydata)

  vars n mean   sd median trimmed  mad min max range skew kurtosis   se

1   1 6    4 4.94   2       4 0.74   1  14   13 1.31  -0.16 2.02

Calculate the Confidence Interval (CI) with 95% confidence level which imply 97.5th percentile of the student t distribution at the upper tail:

> mydata = c(1,2,2,2,3,14)

> E = qnorm(0.975)*describe(mydata)$se   # for t distribution it is qt(.975, df=n-1)

> E

[1] 3.952459

> ci = c(-E,E) + describe(mydata)$mean

> ci

[1] 0.04754095 7.95245905

Using student t-distribution:

> t.test(mydata)

        One Sample t-test

data:  mydata

t = 1.9835, df = 5, p-value = 0.1041

alternative hypothesis: true mean is not equal to 0

95 percent confidence interval:

 -1.18383  9.18383

sample estimates:

mean of x

        4

Calculate Regression:

Copy the CSV file "regression.csv" in your home directory

Calculate the co-efficient "b" and the slope "m => AptitudeTestScore"

module load R

R

> aptitude = read.table("regression.csv", sep=",", header=T)

> head(aptitude)

  Student AptitudeTestScore StatisticsGrade

1       1                95              85

2       2                85              95

3       3                80              70

4       4                70              65

5       5                60              70

> reg = lm( StatisticsGrade ~ AptitudeTestScore, data=aptitude )

> reg

Call:

lm(formula = StatisticsGrade ~ AptitudeTestScore, data = aptitude)

Coefficients:

      (Intercept)  AptitudeTestScore

          26.7808             0.6438

Now, fit the curve in the scattered plot.

> par(cex=.8)

> plot(reg)

> abline(reg)

You will see the scatter plot as showed:

Calculate the "Coefficient of Determination (r^2) for the linear regression

> summary(reg)$r.squared

[1] 0.4803218

See the Regression Summary

> summary(reg)

Call:

lm(formula = StatisticsGrade ~ AptitudeTestScore, data = aptitude)

Residuals:

     1      2      3      4      5

-2.945 13.493 -8.288 -6.849  4.589

Coefficients:

                  Estimate Std. Error t value Pr(>|t|)

(Intercept)        26.7808    30.5182   0.878    0.445

AptitudeTestScore   0.6438     0.3866   1.665    0.194

Residual standard error: 10.45 on 3 degrees of freedom

Multiple R-squared:  0.4803,    Adjusted R-squared:  0.3071

F-statistic: 2.773 on 1 and 3 DF,  p-value: 0.1945

Chi-Squared Test:

> observed = c(50,45,5)

> expectedProb = c(0.3,0.6,.1)

chisq.test(observed, p=expectedProb)

        Chi-squared test for given probabilities

data:  observed

X-squared = 19.583, df = 2, p-value = 5.592e-05

Logistic Regression

Plot:

data = read.table("sample.csv", sep=",", header=T)

plot(data$math,data$hon,main="Regression Plot",xlab="Math Score",ylab="Honor")

Logit function:

Case I: no predictor

mylogit = glm(hon ~ 1, family=binomial(link=logit), data=data) > summary(mylogit) Call: glm(formula = hon ~ 1, family = binomial(link = logit), data = data) Deviance Residuals: Min 1Q Median 3Q Max -0.7497 -0.7497 -0.7497 -0.7497 1.6772 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -1.1255 0.1644 -6.845 7.62e-12 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 222.71 on 199 degrees of freedom Residual deviance: 222.71 on 199 degrees of freedom AIC: 224.71 Number of Fisher Scoring iterations: 4

Case II: one dichotomous predictor

> mylogit = glm(hon ~ female, family=binomial(link=logit), data=data) > summary(mylogit) Call: glm(formula = hon ~ female, family = binomial(link = logit), data = data) Deviance Residuals: Min 1Q Median 3Q Max -0.8337 -0.8337 -0.6431 -0.6431 1.8317 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -1.4709 0.2690 -5.469 4.53e-08 *** female 0.5928 0.3414 1.736 0.0825 . --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 222.71 on 199 degrees of freedom Residual deviance: 219.61 on 198 degrees of freedom AIC: 223.61 Number of Fisher Scoring iterations: 4

Case III: one continuous predictor

> mylogit = glm(hon ~ math, family=binomial(link=logit), data=data

Plotting with logistic Regression:

> plot(data$math,data$hon,main="Math Score Vs Honors",xlab="Math Scores",ylab="Probability of Honors") > curve(predict(mylogit,data.frame(math=x),type="resp"),add=TRUE)

Note: add=True => adding on the plot

Case IV: all predictors

> mylogit = glm(hon ~ female + math + read, family=binomial(link=logit), data=data)

or

> mylogit = glm(hon ~ ., family=binomial(link=logit), data=data)

Odd Ration and CI:

> exp(cbind(OR = coef(mylogit), confint(mylogit))) Waiting for profiling to be done... OR 2.5 % 97.5 % (Intercept) 0.2297297 0.1312460 0.3792884 female 1.8090145 0.9362394 3.5929859

Count:

> library(plyr)

> count(data,'female') female freq 1 0 91 2 1 109 > count(data,'hon') hon freq 1 0 151 2 1 49

> xtabs(~ hon + female, data=data) female hon 0 1 0 74 77 1 17 32

References:

[1] Wikipedica - wikipedica.com

[2] Log of Odds: http://www.ats.ucla.edu/stat/mult_pkg/faq/general/odds_ratio.htm

[3] https://www.mathsisfun.com

[4] Linear Regression Tutorial - http://stattrek.com/regression/regression-example.aspx?tutorial=ap

[5] Logistic Regression - http://www.ats.ucla.edu/stat/mult_pkg/faq/general/odds_ratio.htm

[6] Maximum Likelihood Estimation (MLE) - http://www.wright.edu/~thaddeus.tarpey/ES714glm.pdf

[7] MLE (Logistic) - http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf