Testing Whether Trading Strategy Profits Reflect Skill or Luck
Summary
The document discusses ways to assess whether a strategy’s live profits reflect a repeatable signal or chance, especially when market volatility is large relative to the observed gains. One proposed test, attributed to Alfred Cowles, preserves the trader’s number of position changes but assigns those changes random times. Comparing the strategy’s performance with many such simulated records shows how it ranks against timing generated by chance.
A separate approach measures portfolio exposure to market and other systematic risk factors, then examines the residual profit and loss after accounting for those exposures. The discussion also suggests controlling unwanted factor exposures during portfolio construction. Another answer recommends out-of-sample testing and cautions that searching many random strategies can produce an apparently strong performer by chance. The document gives conceptual suggestions rather than a worked analysis; it does not specify statistical thresholds, simulation design details, or how to assess a live record lasting only one month.
Key ideas
- A random-timing test can preserve the observed number of position changes while testing alternative entry and exit times.
- Strategy performance can be compared with the distribution of results from these randomized records.
- Factor analysis can separate systematic portfolio exposures from residual strategy returns.
- Portfolio construction can limit exposure to risk factors the strategy is intended to avoid.
- Out-of-sample evaluation helps address the risk of selecting a strong result from many trials.
Tags
Full text
# How likely it is that a strategy profits are explained by luck? # How likely it is that a strategy profits are explained by luck? I want to evaluate a trading strategy. My goal is not to compare it with other strategies, but rather to determine how likely it is that the profits are generated from the strategy itself rather than luck. Indeed, the market volatility is very high and of same order of magnitude than the strategy profits. My starting point would be to compare the strategy with a large number of random strategies that are backtested for the same time period. By random strategy I mean a strategy where assets are randomly bought and sold. I can then determine the likely-hood that the strategy profits are explainable by luck. Could you please comment on this test or propose alternative tests? Many thanks Edit: The strategy has been running live for one month. Profits refers to live profits, not backtest ## Answer by nbbo2 (score 3, accepted) https://quant.stackexchange.com/a/42229 Something along these lines is known as the Cowles Test, suggested by Alfred Cowles in 1933 (Can stock market forecasters forecast? https://cowles.yale.edu/sites/default/files/files/pub/misc/cowles-forecasters33.pdf . See page 318 "statistical interpretation of the results." The strategies in question consisted of getting into the market (go long) at certain times and getting out (sell) at other times. He compiled random records to compare to the record of actual traders as follows: First determine how many times the trader changed his position (switched from long to flat or vice versa) during the test period. Then generate as many changes of position as the trader, but at random times during the test period. Evaluate the performance of the random strategy. The human trader's performance can then be ranked along the artificial traders to see if he performed better than randomly. Due to lack of suitable random number generators in 1933 Cowles actually used pieces of paper picked out of a hat to generate the random switching times (!). Today with computers the Cowles Test is much easier to perform. ## Answer by Ezy (score 3) https://quant.stackexchange.com/a/42231 I believe that by "luck" you mean that you want to check if you can attribute the pnl of your strategy to something else than the "alpha" that it's trying to capture. The standard way of doing this is by using standard market factors (such as Barra's standard risk model for equities say https://www.msci.com/www/research-paper/barra-s-risk-models/014972229 ) and calculate the beta of your portfolio against all those factors. The residual pnl will be what is left of your strategy after you account for those factor returns. For example the most simple one is to assess how directional your strategy is by calculating the market beta of your portfolio which will tell you how much of your strategy pnl is coming from the overall market return. But there are plenty other such risk factors which you want to account for (momentum, volatility, big cap vs small cap etc...) Once this is done the natural next step for you is to "control a priori" the exposure to those factors that you want to be orthogonal to. That's one of the core features of the portfolio optimization that you perform in statistical arbitrage: given a set of alphas what is the optimal portfolio that maximizes exposure to them which minimizes transaction costs, balances with risk preference and is orthogonal to a collection of undesired risk factors. ## Answer by XYQ (score 1) https://quant.stackexchange.com/a/42228 To see if a strategy really work or not, you need to run an out of sample backtest. Compare with other random strategy will not work. For example, you can select a period and generate random strategy, there will be always one with relative high sharpe, which beat all the other strategy. ## Answer by Charles Fox (score 0) https://quant.stackexchange.com/a/42226 There is some research on ranking a strategy against randomly generated positions. See "A uniformly distributed random portfolio" by Kim and Lee 2016
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.