Estimating Sample Size for Short-Horizon Strategy Validation

# How much data is needed to validate a short-horizon trading strategy?


# How much data is needed to validate a short-horizon trading strategy?












Suppose one has an idea for a short-horizon trading strategy, which we will define as having an average holding period of under 1 week and a required latency between signal calculation and execution of under 1 minute. This category includes much more than just high-frequency market-making strategies. It also includes statistical arbitrage, news-based trading, trading earnings or economics releases, cross-market arbitrage, short-term reversal/momentum, etc. Before even thinking about trading such a strategy, one would obviously want to backtest it on a sufficiently long data sample.

How much data does one need to acquire in order to be confident that the strategy "works" and is not a statistical fluke? I don't mean confident enough to bet the ranch, but confident enough to assign significant additional resources to forward testing or trading a relatively small amount of capital.

Acquiring data (and not just market price data) could be very expensive or impossible for some signals, such as those based on newer economic or financial time-series. As such, this question is important both for deciding what strategies to investigate and how much to expect to invest on data acquisition.

A complete answer should depend on the expected Information Ratio of the strategy, as a low IR strategy would take a much longer sample to distinguish from noise.

## Answer by chrisaycock (score 50, accepted)

https://quant.stackexchange.com/a/1904

Consider the standard error, and in particular the distance between the upper and lower limits:

\begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation}

Using the formula for standard error, we can solve for sample size:

\begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\right)^{2} \end{equation}

where $s$ is the measured standard deviation, which you already have from your IR calculation.

High-frequency Example

I was testing a market-making model recently that was expected to return a couple basis points for each trade and I wanted to be confident that my returns were really positive (ie, not a fluke). So, I chose a distance of 3 bps $(\Delta = .0003)$. My sample's measured standard deviation was 45 bps $(s = .0045)$. For a confidence interval of 95% $(\alpha = 1.96)$, my sample size needs to be $n = 3458$ trades. I would have picked a tighter distance if I had been simulating this model, but I was trading live and I couldn't be too choosy with money on the line.

Low-frequency Example

I imagine that for a low-frequency model that was expected to return 1.5% per month, I'd want maybe 1% as the distance $(\Delta = .01)$. If the hoped-for Sharpe ratio were 3, then the standard deviation would be 1.7% $(s = .017)$, which I came-up with by backing-out the monthly returns. So for a confidence interval of 95% $(\alpha = 1.96)$, I'd need 45 months of data.

## Answer by Michael WS (score 4)

https://quant.stackexchange.com/a/1913

I would also note that you need to watch out for correlations between data points. (EG ,if you have a data point proving this works for oil company x. Another data point for oil company y may not actually count as separate.)

If you are looking at 5 day holding periods, why not just grab all the EOD data that you can as well.EOD data is obviously not tradeable but can be used as a sanity check for long term trading strategy returns when you do not actually have the data.

## Answer by Jon Grah (score 4)

https://quant.stackexchange.com/a/32595

Surprisingly, no one mentioned the need to ensure that the trading strategy can survive different types of market conditions. Suppose that you use @chrisaycock's formula and came up with 5000 trades. Well, you could make 5000 trades in a few hours or a 1-2 months with the original criteria. Putting aside amount of data to test on, it could just so happen that the time period you selected to test and collect 5000 trades worth of data 'fit' your strategy well. What about 5001th trade or 6000th trade? I would be asking how would you have handled a 'flash crash' or sudden change in product value? A regulator report that a key license required to legally operate the primary business function has been suspended or revoked effective immediately?

A more precise answer in my opinion and experience is to make sure that you sample enough tick data to cover a variety of market conditions. Pay close attention to volatility surrounding sudden events as well as economic calendar announcements (mainly sharp moves in prices). The algo would need to know how to adjust to handle these moves. So you might initially run it over a couple of years of tick data and note these sharp moves and different flavors of volatility. Then on subsequent runs you can focus on the sharp moves first, then replay on the remaining data to further validate the strategy.

## Answer by Ishan Shah (score 0)

https://quant.stackexchange.com/a/44512

To comprehensively validate a short horizon strategy you will need following

- Data





It is very difficult to find a strategy which works all the time but after doing this analysis, you will have clarity on which market conditions and on which instruments it works.