Designing Benchmarks for Algorithmic Trading Backtests
Summary
The document describes a proposed way to compare an algorithm’s backtest with a simple benchmark portfolio. The benchmark buys some or all stocks in a selected universe at the start of the test period, then closes positions at the end, with transaction fees and slippage included. The algorithm is allowed to choose stocks over time, and its costs are intended to be treated similarly.
The author asks whether the algorithm should be optimized for profit above the benchmark and which portfolios and profit calculations make a fair comparison. Options include comparing aggregate holdings, individual positions, or the stocks selected by the algorithm, and handling positions that remain open at the end. No answers or results are provided. A sound comparison would also need consistent capital allocation, exposure, timing, and treatment of open positions; the document leaves those choices unresolved.
Key ideas
- The proposed benchmark buys a fixed stock set at the start and closes it at the end of the period.
- The algorithm selects positions over time, with transaction costs and slippage included in both runs.
- The author considers optimizing excess performance over the benchmark instead of absolute profit.
- The document raises unresolved choices about aggregate versus position-level comparisons and open positions.
Tags
Full text
# Backtesting benchmark / control test design # Backtesting benchmark / control test design I am designing a backtesting system to test an algorithm. I'd like to have a benchmark / control test results to compare to the results of a custom algorithm. A limited "set" of stocks is selected to invest in. The algorithm can choose to buy one or more stocks. In an effort to have a benchmark to measure against, the backtest system can: - Buy some or all of the stocks on the first day of the test period. - Sell all of the positions of the set at the end of the test period. - Include transaction fees and slippage costs during the test period. To test the algorithm, the backtest system can: - Allow the algorithm to choose to buy one or all stocks on the first test day. - Loop the algorithm through time for until the last day of the testing period. - Include similar transaction fees and slippage costs during the test period. The profit percentage of the "set test" and "algo test" are recorded. The "set test profit" is deducted from the "algo test profit" to rank the test. QUESTION 1: Should the algorithm be optimized to outperform the benchmark test (rather than simply optimizing for the algorithm's profit)? QUESTION 2: What test results should be compared? - A. Buying and selling the entire set VS. ALL the positions the algo bought. - B. Buying and selling the entire set VS. EACH individual position the algo bought.[1] - C. Buying and selling the set of stocks the blackbox bought VS. The algo results. - D. Buying THE stock the algo bought for the position VS. the algo position result. QUESTION 3: How should the profits of the tests be calculated? - A. Buying and selling all positions on both the "set test" and "algo test".[2] - B. Buying and selling the "set test" VS. Allowing the algo to keep unsold stocks. - C. Buying the set and selling stocks that end the algo test sold VS. algo results.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.