Testing Stock Trading Hypotheses with Random Portfolios and Out-of-Sample Data

Summary

The document outlines a way to evaluate a hypothesis that selected stocks drift after a measured deviation from a market index. First specify the trading rules: what triggers entry and how positions are closed. Apply those rules to historical data to calculate strategy profit and loss over the chosen period. The answer suggests repeatedly running the same process with randomly selected stocks, then comparing the strategy’s result with the distribution from those random portfolios. A result outside the upper tail would be more encouraging than one typical of the random selections.

The comparison does not establish that the effect is real. If the idea was developed by examining the same market history, evaluating it on that history risks fitting the hypothesis to its source data. A genuinely out-of-sample evaluation helps address that issue, but still cannot guarantee future performance because market behavior changes. The document gives a testing framework rather than empirical evidence that the proposed stock pattern works, and it does not specify transaction costs, risk controls, or a formal test design.

Key ideas

• Define entry triggers and exit rules before calculating historical profit and loss.
• Compare the strategy with repeated portfolios of randomly selected stocks.
• A result that is not in the upper tail of the random comparison is weak evidence for the strategy.
• Testing on data used to form the hypothesis can produce misleading results.
• Out-of-sample performance does not ensure the effect will persist as markets change.

Tags

# Testing a simple stock market trading hypothesis?


# Testing a simple stock market trading hypothesis?












I have a simple stock market trading hypothesis ...

Something along these lines:

> Based on a certain deviation from a stock market index over a 30-day period, 5 certain stocks tend to drift in a particular direction during the next 30-day period.

I'd like to run this hypothesis on historical data to see if there's any truth to this.

How do I go about?

## Answer by Patrick Burns (score 3, accepted)

https://quant.stackexchange.com/a/2368

In terms of technology, I would suggest R.

In terms of specific actions, you need to decide what to do when the trigger event occurs and you need to decide how to close the positions. Given that, you can then determine your profit and loss on the strategy over the data period.

You can use the random portfolio idea by running your strategy a number of times except use 5 randomly selected stocks in place of your specific 5 stocks. You will then get a distribution of profit and loss from the random stocks to compare against the result from your real strategy. If your strategy is not in the upper tail, then don't do your strategy.

But even if your strategy does look good in that test, that doesn't mean it is a real phenomenon. You got your idea from looking at market data. You don't want the test to be based on data that gave you the idea.

And of course even if your strategy passes a truly out-of-sample test, markets can (and do) change.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.