Market-Regime Filters and Data Selection in Backtesting

# Is it statistically valid to ignore "irrelevant" stock prices during backtesting?


# Is it statistically valid to ignore "irrelevant" stock prices during backtesting?












As an amateur trader, I have reasonable success with my current end-of-day trading systems.

All my systems are based on at least ONE year of "in-sample" data for backtesting, followed by 6-months of "out-of-sample" data.

So my question is: Could I take TWO years of "in-sample" data, but ignore any data for periods in which I would not have traded due to the market conditions?

(I used the "data" tag, and searched back to November 2011. I searched all posts for "Historical data".I could not find any related posts.)

Example: Assume I want to trade a "long-only" system, in an "upward-trending" market.

Assume the chart for the previous two years looked like a "W". The first 6 months trends down, then 6 months up, then down, then up.

Can I simply ignore the first and third 6-month periods, and use only the second and fourth 6-month periods?

Reason: If the market was trending down, I would not trade. So, why should my system be based on any previous market data that was trending downwards?

Method:

- In the "W" example above, copy and paste the second 6-month data into Excel.

- Ignore the third 6-month data.

- Copy and paste the fourth 6-month data into Excel.

- But adjust the entire second batch of data upwards so that it joins seamlessly to the end of the first batch. (Possibly by making the first price of the second batch equal to the last price of the first batch.)

- So, the result would be a 1-year data series that only trended upwards.

I would be most grateful for any comments.

(Added after Nathan S's answer below. My addition is too long to be added as a comment to Nathan S's answer.)

> It all depends to me on whether your system can categorize a price series as "upward trending."

My first impression was that this is a valid point. So I was about to defend my trend indicator (TE). But, on reflection, I now think it's a distraction. It occurred to me that people such as engineers, IT troubleshooters, and people who conduct human trials always attempt to isolate the "common factor(s)" between two sets of experiments. After that, any differences between the results of the experiments will depend only on differences between the components.

In my case, I think the TE (good or bad) is a common factor. It has already been optimized on a watchlist using 3 years of "undoctored" market data. I use that TE for any system based on that watchlist. (For other watchlists, I have different TEs.)

So the TE will be common to both sets of experiments (doctored data versus undoctored data). For the doctored data, obviously I need to extend the starting point further back in time in order to capture the required number of bars (days) of doctored data.

Here are some statements that I have seen regarding the "selection" or "cherrypicking" of data for backtesting. Unfortunately, I have no links, because I tended to agree with all these statements. (I'm more likely to keep links for statements that I disagree with, for later research.)

Don't assume that a system that works with one of the following will work with the other:

- US stocks versus Australian stocks.

- A portfolio of Financials versus a portfolio of Utilities.

- A bull market versus a bear market.

I think it was the final statement that made me "invent" my own statement: Don't optimize an "uptrend" system using down-trending data.

## Answer by Nathan S. (score 1)

https://quant.stackexchange.com/a/16891

It all depends to me on whether your system can categorize a price series as "upward trending." I want objective, programmed rules. If my system includes a market condition filter then not only can I ignore cases that would not pass it, but I must ignore them to measure the system.

If I invent the rule to optimize performance that ran cold in some tested periods, (Not saying that's how your market conditions filter was concocted, but I have done this) then I'm very wary and even more skeptical with out of sample.

## Answer by brian (score 1)

https://quant.stackexchange.com/a/16916

You have to be careful when saying "I would have done this". It's too easy in backtesting to make this mistake. From your description of the data, you have no way of knowing it was in a downtrend, until the downtrend was over or ,at least already in full swing. Nathan S's answer and radpin's comments are exactly what you have to do.

I didn't answer just to tell you to listen to other answers but to give an example of things I do. I know that if a market closes at noon one day, I wouldn't have traded and I would know that in advance. I can also make a volume cutoff so I don't trade over the xmas holidays for example. The system only knows this after the fact but I can safely assume I know when the holidays are, so I ignore discard those results after the fact.