Why Large Samples Can Make Kolmogorov–Smirnov Tests Reject
Summary
The document considers a Kolmogorov–Smirnov test applied to returns from simulated geometric Brownian motion. The question describes simulating many terminal asset prices, converting them to returns, fitting distribution parameters, and testing the returns against a lognormal distribution. The response focuses on sample size rather than diagnosing the simulation or implementation.
With 100,000 observations, the test has high power to detect even small departures from the specified distribution, so a very small p-value does not by itself show that the simulation is fundamentally wrong. The response suggests that a similarly large sample drawn directly from a lognormal distribution may also be rejected. This is a caution about interpreting significance tests, not a validation of the supplied code or data handling. The discussion does not examine whether simple returns are the appropriate quantity to test, how parameters should be estimated for a goodness-of-fit test, or what alternative diagnostic would be suitable.
Key ideas
- A Kolmogorov–Smirnov test can detect small distributional deviations when the sample is very large.
- A small p-value in a large sample does not by itself identify a coding error.
- The response recommends interpreting statistical significance in light of sample size and the chosen significance threshold.
- The answer does not verify the simulation code or resolve the distribution-fitting details.
Tags
Full text
# Perform scipy Kolmogorov-Smirnov Test for lognormal distribution in GBM
# Perform scipy Kolmogorov-Smirnov Test for lognormal distribution in GBM
I am simulating asset prices for n days using GMB with Euler scheme, calculate returns and then perform Kolmogorov-Smirnov test on simulated returns. Code for simulating GBM :
```
def simulate_GBM(mu, sig, asset_price, number_of_days):
delta = 1 / 252
t = np.arange(0, number_of_days / 252, delta)
path = np.zeros(len(t))
path[0] = asset_price
for i in range(0, len(t) - 1):
path[i + 1] = path[i] + path[i] * mu * delta + \
sig * path[i] * sqrt(delta) * np.random.randn()
if path[i + 1] < 0:
print("Zero assset values!")
path[i + 1] = 1
return path
```
Code for Monte-Carlo GBM
```
def perform_GBM_Monte_Carlo(mu, sig, asset_price, num_sim, number_of_days):
simulations = np.zeros(num_sim)
stop_date = number_of_days
for i in range(0, self.num_sim - 1):
simulations[i] = (simulate_GBM(mu, sig, asset_price, num_sim, number_of_days)[stop_date - 1])
return simulations
```
Then I calculate returns and perform test
```
data = perform_GBM_Monte_Carlo(0.05, 0.3, 100, 100000, 252)
returns = np.zeros(len(data))
for i in range(0, len(data) - 1):
returns[i] = data[i]/100 - 1 # initial asset price is 100
lognorm_params = stats.maxwell.fit(returns)
t_stat, p = stats.kstest(returns, 'lognorm', lognorm_params)
```
Theory says that returns must be lognormally distributed, but I have very p_value. Is the problem with GBM simulation or the way I perform K-S test?
## Answer by SmallChess (score 1, accepted)
https://quant.stackexchange.com/a/34050
I haven't checked your code, but ... even if your code is all correct (you have calculated the lognormal returns), your test won't work.
You have 100000 samples, this is a large number. With this large sample size (and huge statistical power), the KS-test will reject anything. If you don't believe me, try to draw 100000 lognormal distribution directly from Python, your KS-test will still reject you for very low p-value.
Statistical test is meaningless for a large sample size, unless you want very low level of significance.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.