Skip to content
All library documents

Why Automated Trading Backtests Can Fail to Predict Live Performance

Article Quant Q&A · Author: Alexander Chervov
Read the original

Summary

The document asks whether automated trading strategies tend to make or lose money and seeks comparative tests of simple strategies on historical data. It offers no such benchmark study or measured performance results. Instead, one response describes a pattern observed while examining strategies once offered for rent: developers may optimize a strategy on past data, then run it on a demo account and begin selling subscriptions after a favorable short live period. The response warns that this can create a misleading impression of reliability, especially when leverage is high.

A second response adds that a strategy’s market edge may fade as conditions change, so an algorithm may need to be revised or restarted. These are cautions and personal observations, not a systematic evaluation of average robot returns. The document gives no defined sample, testing protocol, transaction cost assumptions, or evidence comparing strategies across markets or time periods. It therefore cannot establish typical profitability or supply the requested benchmarks; it mainly highlights overfitting, weak verification, leverage, and changing market conditions as risks to investigate.

Key ideas

  • A strategy optimized on historical data may perform poorly once it is traded live.
  • A brief favorable demo period does not establish durable profitability.
  • High leverage can make short term results look compelling while increasing risk.
  • Market conditions can change, causing a previously useful trading edge to decay.
  • The document offers warnings and observations rather than comparative backtest evidence.

Tags

BacktestingRisk management
Full text
# Automatic trading strategies - what are benchmarks for PL on serious backtesting?


# Automatic trading strategies - what are benchmarks for PL on serious backtesting?












There are plenty books and sites which offer automatic trading strategies (robots) and claim they are very profitable. On the other hand many people do not believe in such things, saying that: if there would be "graal" that generates stable profit, then the details would be kept in secret, and many other reasonable doubts in such claims.

So what is the truth ? Does most/average automatic trading system lose money in say one year/ 3 years ? or may be they win 100% in a week :) ?

More seriously: I am interested to find some analysis that would take several trading strategies and test it on various historic data and describe the results. It is better that strategies would be simple/standard and invented by third party in order to avoid conflict of interets.

## Answer by chjortlund (score 1)

https://quant.stackexchange.com/a/15727

As mentioned above, your post is very broad and therefore difficult to answer.

However, I have had a lot of fun trying to reverse engineer some strategies, from back when RentASignal.com was in business. My general conclusion where that the strategies for rent where optimized for a great back test, the developers then set the strategy up for a forward test using an a demo account, and then crossed their fingers.

If they were lucky, their (typically) high leveraged strategy made XXX% return over the next 1 – 2 months, and then started to get paying subscribers.

This is a lucrative and easy business for fraudsters, so watch out, and never buy anything unless there is solid test and verification documentation available (there never is!)

## Answer by Arik Noi (score 0)

https://quant.stackexchange.com/a/15251

Markets are changing all the time and that's is why a specific "edge" is lasting some time and than you will need to restart your algo.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.