Why Backtest Performance Changes Across Market Periods

# Backtesting Period Effect


# Backtesting Period Effect












I am backtesting a stock trading strategy. I tested it over two time periods: 2000-2020 and 2015-2020 and compared the results against a buy and hold strategy. To be clear, I only changed backtesting period and kept all other settings/parameters the same. The equity curve (shown below) is promising over 2000-2020 and looks really bad over 2015-2020.

Q: Am I right that given how sensitive the results are with respect to the changes to the backtesting period, this trading strategy is basically worthless? In other words, a good trading strategy should consistently beat at least buy and hold strategy when backtesting period is changed?

## Answer by demully (score 3)

https://quant.stackexchange.com/a/59850

This is a common problem (with backtesting in general); and one that is easy to lazily generalise; but difficult to pin down in a more forensic manner.

Beating buy-and-hold isn't a definitive test of whether your strategy is "good" or "bad"; but it does offer an intuitive and consistent alternative to measure it against. Measuring the ratio of your strategy against B&H tells you quickly when your strategy made/lost relative to B&H.

Optically, it's quite clear that the 2000-2020 backtest earns it keep by dodging the bear markets of the dotcom and GFC crashes. Those periods were flat in black and down in red. Black trumps Red. And it's equally optically clear that the reverse was true in the second 2015-2020 run. The flat black periods tended to be red-positive after the. If you measured the market (ie B&H) returns in the periods when your strategy was out of the market, I'd be very very surprised if they weren't negative before 2015 (maybe even 2013 or 2011); and positive after then.

Which might be suggestive that the payoffs associated with "buying the dip" changed between the two regimes. One can point fingers at the central banks and foam at the mouth ranting about QE; but one can never quite causally prove anything about its market impacts.

One can equally worry about backtests, like armies, always "trying to fight the last war", taking a Maginot Line to a tank fight. The point being that everyone's backtests now know about the GFC, the "risk-on,risk-off" regime that followed it, and the "liquidity-on, liqduity-off" regime that ensued after circa 2013. They're all optimising to the same limited sample.

I recall building a strategy like this circa 2005, that got bottled up into a range of structured products. Back then, all of definitions of what "disaster" looked like were heavily, heavily, biased to the 97-98 Asia/Russia/LTCM crisis, probably even more than to dotcom. However constructed, any model that flunked that scenario would be rejected out of hand without further examination. The "easy" way to do that was to "get out when things turn south" (like your strategy optically appears to do); but then "get back in when things turn so bad they can't really be that bad". Do that at -2 sigma, you made a lot of money with a lot of vol; at 3 sigma, you dodged most of the downturn with a huge roundtrip reduction in vol... Which is where most these strategies hit landmines in 2008. They bought the dip in Q4-08 after Lehmans, and oops. But if your model was identical, but configured to try to short the downswings, then you didn't buy the dip. When things got that bad, you sat in "I don't know" cash. Then you looked like a genius (selling on the way down before closing shorts near to the bottom). Even when everybody's model was virtually identical to all of the others! It was just configured to implement differently to different segments. Those segments being different in 2008 to 1998, as simple as that. Hero and villain alike were all collectively over-fitted to 1998 then.

Fast forward, and 2008 still plays the same role today as 1998 used to play before 2008. It is "the big one" that any model has to navigate. And just as 1998 biased everyone's reactions to 2008, everyone's memories of 2008 has the same potential to bias them today.

Proof of concept would be to optimise your strategy for the 2015-20 and see how it did during the dotcom and GFC episodes. I'll bet that the "genius" market-timing strategy of the last 5 years would have been taken to the cleaners then... because it won't be so biased by 2008 to cause it to ignore a very different market reaction function since then. Whether the reaction function is different because we all remember 2008 is, of course, unprovable.

But if you have a 2000-15 model that is neg-alpha in 2015-20; and a 2015-20 model that is neg-alpha in 2000-15, that would from basic first principles suggest that the market's reaction function was different between the two periods in question. Which is the central problem facing all market-timing strategies!

## Answer by MainCom (score 1)

https://quant.stackexchange.com/a/59822

The strategy doesn't look promising to me. It looks 'good' in the first half period simply because it avoided the .com-bubble and GFC. Whether it is worthless depends on whether the strategy can be improved. You could look into the trades in more details to find out why 2002-2007 is profitable while 2013-2020 is not.

It is not necessary for a good strategy to beat buy and hold. We should also look at other measurements such as Sharpe, drawdown, etc.

## Answer by user42108 (score 0)

https://quant.stackexchange.com/a/59843

[is] this trading strategy is basically worthless?

First impression was it was overfitted and/or the alpha decayed but you haven't provided enough info to answer the question.

In other words, a good trading strategy should consistently beat at least buy and hold strategy when backtesting period is changed?

I'm confident that if you take the track records of successful equity investors, you will find periods where they underperform "the market" for some time. Not 'consistently beat[ing] buy and hold' is not sufficient to abandon/rule out a strategy.

EDIT: Chart I had in mind to illustrate my point. I don't remember the source.