1.11. Non-Parametric test for Hypothesis Testing: Komolgorov-Smirnov

Kolmogorov–Smirnov Test is a completely efficient manner to determine if two samples are significantly one of a kind from each other. It is normally used to check the uniformity of random numbers. Uniformity is one of the maximum important properties of any random number generator and the Kolmogorov–Smirnov check can be used to check it [1]. 

The Kolmogorov–Smirnov test is versatile and can be employed to evaluate whether two underlying one-dimensional probability distributions vary. It serves as an effective tool to determine the statistical significance of differences between two sets of data.

Kolmogorov Distribution

The Kolmogorov distribution, often denoted as D, represents the cumulative distribution function (CDF) of the maximum difference between the empirical distribution function of the sample and the cumulative distribution function of the reference distribution. 

The probability distribution function (PDF) of the Kolmogorov distribution itself is not expressed in a simple analytical form. Instead, tables or statistical software are commonly used to obtain critical values for the test. The distribution is influenced by sample size, and the critical values depend on the significance level chosen for the test.

where: 

• n is the sample size. 

• x is the normalized Kolmogorov-Smirnov statistic. 

• k is the index of summation in the series.

The next code is designed to generate and plot the probability density function (PDF) of the Kolmogorov-Smirnov distribution for a specified sample size n. These are the following key points of the code:

1. Import Libraries: The code imports necessary libraries: numpy for numerical operations, matplotlib.pyplot and seaborn for plotting, and scipy.stats for statistical functions.

2. Set Sample Size: A variable n is set to 10, indicating the sample size for which the Kolmogorov-Smirnov distribution will be evaluated.

3. Generate Uniformly Distributed Random Values: An array x of 1000 random values is generated from a uniform distribution between 0 and 1 using np.random.uniform.

4. Calculate PDF of Kolmogorov-Smirnov Distribution: The PDF of the Kolmogorov-Smirnov distribution for sample size n is calculated at the points specified in x using stats.kstwo.pdf.

5. Plot the Results: 

• A line plot is created using seaborn to visualize the PDF of the Kolmogorov-Smirnov distribution. The x-axis represents the uniformly distributed random values, and the y-axis represents the corresponding PDF values.

• The plot includes a title indicating the sample size n, and labels for the x and y axes.

# Calculate samples

n = 10

x = np.random.uniform(0, 1, 1000)

y = stats.kstwo.pdf(x, n = n)

plt.figure(figsize = (8,5))

sns.lineplot(x = x, y = y)

plt.title(f"Kolmogorov-Smirnov Distribution for en={n}")

Text(0.5, 1.0, 'Kolmogorov-Smirnov Distribution for en=10') 

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

from scipy import stats

# Calculate samples

n = 10

x = np.random.uniform(0, 1, 1000)

y_cdf = stats.kstwo.cdf(x, n=n)

# Set seaborn style

sns.set(style="whitegrid")

# Create the figure and axes

plt.figure(figsize=(10, 6))

# Plot the CDF line

sns.lineplot(x=x, y=y_cdf, color='green', label='CDF')

# Title and labels

plt.title(f"Kolmogorov-Smirnov Cumulative Distribution for n={n}", fontsize=16)

plt.xlabel('x', fontsize=14)

plt.ylabel('Cumulative Probability', fontsize=14)

# Show the legend

plt.legend()

# Show the plot

plt.show()

import numpy as np

import scipy.stats as stats

import matplotlib.pyplot as plt

# Generate a small sample size (e.g., 25 samples) from a normal distribution

np.random.seed(0)  # For reproducibility

sample_size = 25

sample = np.random.normal(loc=0, scale=1, size=sample_size)

# Perform the Kolmogorov-Smirnov test for normality

d_statistic, p_value = stats.kstest(sample, 'norm')

# Print the KS test result

print(f"KS Statistic: {d_statistic}")

print(f"P-Value: {p_value}")

# Plot the empirical distribution function (EDF) of the sample

sorted_sample = np.sort(sample)

y_vals = np.arange(1, sample_size + 1) / sample_size

# Plot the cumulative distribution function (CDF) of the reference normal distribution

x_vals = np.linspace(min(sample), max(sample), 100)

cdf_vals = stats.norm.cdf(x_vals)

# Plotting

plt.figure(figsize=(10, 6))

plt.step(sorted_sample, y_vals, where='post', label='Empirical CDF')

plt.plot(x_vals, cdf_vals, label='Reference Normal CDF', color='red')

# Highlight the KS statistic on the plot

# Find the point of maximum difference

d_max_index = np.argmax(np.abs(y_vals - stats.norm.cdf(sorted_sample)))

d_max = np.abs(y_vals[d_max_index] - stats.norm.cdf(sorted_sample[d_max_index]))

plt.plot([sorted_sample[d_max_index], sorted_sample[d_max_index]],

         [stats.norm.cdf(sorted_sample[d_max_index]), y_vals[d_max_index]],

         'k--', label=f'KS Statistic = {d_statistic:.3f}')

# Adding labels and legend

plt.xlabel('Sample Values')

plt.ylabel('Cumulative Probability')

plt.title('Kolmogorov-Smirnov Test for Normality')

plt.legend()

plt.grid()

# Show plot

plt.show()

KS Statistic: 0.26842179992563575

P-Value: 0.044235532757121 

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

from scipy.stats import ks_2samp

# Set the seed for reproducibility

np.random.seed(42)

# Generate two sample datasets

sample1 = np.random.normal(0, 1, 100)

sample2 = np.random.normal(0.5, 1.5, 120)

# Perform the Kolmogorov-Smirnov test

ks_statistic, p_value = ks_2samp(sample1, sample2)

# Print the results

print(f"Kolmogorov–Smirnov Statistic: {ks_statistic}")

print(f"P-value: {p_value}")

# Decision based on p-value

alpha = 0.05

if p_value < alpha:

    print("Reject the null hypothesis. The two samples come from different distributions.")

else:

    print("Fail to reject the null hypothesis. There is not enough evidence to suggest different distributions.")

# Plot the histograms with KDE

plt.figure(figsize=(12, 8))

sns.histplot(sample1, bins=20, kde=True, color='b', label='Sample 1')

sns.histplot(sample2, bins=20, kde=True, color='g', label='Sample 2')

plt.legend()

plt.title('Histogram and KDE of Sample Distributions')

plt.xlabel('Value')

plt.ylabel('Frequency')

plt.show()

# Calculate ECDF for both samples

def ecdf(data):

    """Compute ECDF for a one-dimensional array of measurements."""

    n = len(data)

    x = np.sort(data)

    y = np.arange(1, n+1) / n

    return x, y

# Get ECDFs

x1, y1 = ecdf(sample1)

x2, y2 = ecdf(sample2)

# Plot the ECDFs

plt.figure(figsize=(12, 8))

plt.step(x1, y1, where='post', label='ECDF Sample 1', color='b')

plt.step(x2, y2, where='post', label='ECDF Sample 2', color='g')

# Highlight the KS statistic

d_max = np.max(np.abs(np.interp(x1, x2, y2) - y1))

plt.plot([x1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))], x1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))]],

         [y1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))], np.interp(x1, x2, y2)[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))]],

         'k--', label=f'KS Statistic = {ks_statistic:.3f}')

# Adding labels, title, and legend

plt.xlabel('Sample Values')

plt.ylabel('Cumulative Probability')

plt.title('Empirical Cumulative Distribution Functions (ECDF)')

plt.legend()

plt.grid()

plt.show()

Kolmogorov–Smirnov Statistic: 0.35833333333333334

P-value: 9.93895980740741e-07

Reject the null hypothesis. The two samples come from different distributions. 

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

from scipy.stats import ks_2samp

# Set the seed for reproducibility

np.random.seed(42)

# Generate two sample datasets

data1 = np.random.normal(7, 2, 100)  # Normal distribution

data2 = np.random.lognormal(2, 0.2, 100)  # Log-normal distribution

# Perform the Kolmogorov-Smirnov test

ks_statistic, p_value = ks_2samp(data1, data2)

# Print the results

print(f"Kolmogorov–Smirnov Statistic: {ks_statistic}")

print(f"P-value: {p_value}")

# Decision based on p-value

alpha = 0.05

if p_value < alpha:

    print("Reject the null hypothesis. The two samples come from different distributions.")

else:

    print("Fail to reject the null hypothesis. There is not enough evidence to suggest different distributions.")

# Plot the histograms with KDE

plt.figure(figsize=(12, 8))

sns.histplot(data1, bins=20, kde=True, color='b', label='Data 1 (Normal)')

sns.histplot(data2, bins=20, kde=True, color='g', label='Data 2 (Log-normal)')

plt.legend()

plt.title('Histogram and KDE of Sample Distributions')

plt.xlabel('Value')

plt.ylabel('Frequency')

plt.show()

# Calculate ECDF for both samples

def ecdf(data):

    """Compute ECDF for a one-dimensional array of measurements."""

    n = len(data)

    x = np.sort(data)

    y = np.arange(1, n+1) / n

    return x, y

# Get ECDFs

x1, y1 = ecdf(data1)

x2, y2 = ecdf(data2)

# Plot the ECDFs

plt.figure(figsize=(12, 8))

plt.step(x1, y1, where='post', label='ECDF Data 1 (Normal)', color='b')

plt.step(x2, y2, where='post', label='ECDF Data 2 (Log-normal)', color='g')

# Highlight the KS statistic

d_max = np.max(np.abs(np.interp(x1, x2, y2) - y1))

plt.plot([x1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))], x1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))]],

         [y1[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))], np.interp(x1, x2, y2)[np.argmax(np.abs(np.interp(x1, x2, y2) - y1))]],

         'k--', label=f'KS Statistic = {ks_statistic:.3f}')

# Adding labels, title, and legend

plt.xlabel('Sample Values')

plt.ylabel('Cumulative Probability')

plt.title('Empirical Cumulative Distribution Functions (ECDF)')

plt.legend()

plt.grid()

plt.show()

Kolmogorov–Smirnov Statistic: 0.2

P-value: 0.03638428787491733

Reject the null hypothesis. The two samples come from different distributions.