Skip to content
All library documents

Evaluating Backtests Across Years with Changing Market Conditions

Article Quant Q&A · Author: Sid
Read the original

Summary

The document describes a trader’s concern about a strategy tested on one second snapshot data. The trader reports that results from 2018 and 2019 broadly fit expectations, while tests on 2016 and 2017 show small negative annual returns. They note two possible differences in the earlier period: lower market turnover and less average daily movement, the latter being relevant to the strategy. The trader also says the strategy has been used live on a limited, test basis.

The question is whether to investigate the earlier results further or discard the strategy, and how much weight to give results from several years ago relative to the latest two years. These observations raise the issue of changing market conditions and the comparability of backtest periods. However, the document gives no detailed strategy rules, transaction cost assumptions, bid and ask data coverage for each period, statistical analysis, or live performance record. It presents a diagnostic question rather than evidence that resolves whether the strategy is robust.

Key ideas

  • The trader reports profitable expectations in recent backtests and small negative annual returns in earlier years.
  • Lower turnover and reduced average daily movement are offered as possible explanations for the earlier outcomes.
  • The strategy’s sensitivity to market movement may affect how comparable results across periods are.
  • The document asks how to weigh older backtests against recent results but supplies no analysis resolving that choice.
  • Limited live test trading is mentioned, without detailed performance evidence.

Tags

BacktestingEquitiesStatisticsRisk management
Full text
# Data issue or is there an issue in my observations?


# Data issue or is there an issue in my observations?












I am using 1 sec snapshot data to backtest on 2019 data collected by me. It has both bid and ask.

I purchased 4 years data for the same instrument, sadly this data doesn't have bid-ask. When I run the backtest on 2018 and 2019(I purchased another set to be sure), the results are more or less inline with expectations.

Number of trades generated: 332, 517 respectively.

However 2016 and 2017, are giving small negative returns for the year which as per my original thesis should not happen. I investigated and here are my thoughts:

1- The market turnover on the instruments in question was a half of what it is today.

2- The average daily movement on the instrument has increased quite a lot on the instrument, which is a factor in my strategy.

I have used the strategy live and it seems to work. Just test trades as of now.

I am wondering whether to put the concern regarding 2016, 2017 returns on the side(Investigate them more and more but not throw the strategy away) or back to the drawing board.

How significant should I consider the returns of 4 and 3 years ago compared to last two years?

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.