Handling Overnight Gaps in Mean-Reversion Research
Summary
The discussion addresses whether overnight price gaps should be removed when testing cointegration and calculating signals for a stock mean-reversion strategy. The answer ties data treatment to the strategy’s holding period. For an intraday strategy that opens and closes positions within the trading session, overnight returns are outside the strategy’s exposure and can be excluded from the analysis. For a strategy that holds positions across multiple days, end-of-day prices include overnight moves, so those gaps belong in the data used to assess the strategy.
The example describes daily adjusted closing prices for a cointegration calculation and log returns for profit and loss, with the intended trade potentially lasting several days. The response cautions that there is no universal data-cleaning rule: removing observations also removes information. Researchers should align the sampled returns and price series with when the strategy can hold risk, then judge how any exclusions affect statistical results. The answer does not diagnose the example’s reported out-of-sample gains or establish that the strategy is profitable.
Key ideas
- Whether to include overnight returns depends on when the strategy holds positions.
- Intraday strategies can exclude overnight moves if they have no exposure during those hours.
- Multi-day strategies should account for overnight gaps included in end-of-day prices.
- Removing data discards information and can affect statistical conclusions.
- Data cleaning should reflect the strategy’s actual trading horizon and exposure.
Tags
Full text
# Mean-reversion strategy with overnight gaps
# Mean-reversion strategy with overnight gaps
When using stocks as time series data, it is common to encounter large overnight gaps, sometimes because of earnings, other times because of press releases. So, how to correctly account for this feature when checking for cointegration and calculating the z-score for mean reversion? Naively, even a simple triplet UNH-VST-COP would be having large returns out of sample, so I must be missing something here.
Edit: As suggested by @lehalle, below I've added the code which computes the weights using the Johansen method. Notice that (at least, according to my understanding), the Johansen method needs the close prices as inputs (not the returns). On the other hand, the PnL is indeed computed as log return `np.log(p[closeArrayName]).diff()`. In these log returns, overnight gaps are present, hence my question. This is not an intraday strategy, in the sense that a position is opened at the close of a given day and held, classically, until it mean returns, which may occur several days later. As data source, Yahoo finance (`yfinance`) was used and `f` in the code below is a Pandas dataframe file containing each individual ticker (for example, `UHN.h5`).
```
def assembleModelData(symbols, start, end, window, timeframe):
""" Assembles the `modelData` dataframe. """
closeArrayName = 'Adj Close'
# start from empty
modelData = pd.DataFrame()
for s in symbols:
# load
p = pd.read_hdf(f, key = 'data')
# select the time period
p = p.loc[ start : end ]
# compute log prices and stack side by side and rename columns appropriately
modelData = pd.concat([modelData, pd.DataFrame({s: np.log(p[closeArrayName])})], ignore_index = False, axis = 1)
# drop NaN
modelData.dropna(inplace = True)
return modelData
symbols = ['UNH', 'VST', 'COP']
start = '2022-01-01 00:00:00'
end = '2024-05-01 00:00:00'
timeframe = '1d'
window = 30
modelData = assembleModelData(symbols, start, end, window, timeframe)
for i in range(0, len(modelData) - window + 1):
modelDataSub = modelData.iloc[i:window + i].copy()
# cointegration test
j = coint_johansen(modelDataSub, det_order = 0, k_ar_diff = 1)
weights = list(j.evec.T[0] / j.evec.T[0][0])
```
```
## Answer by THATS MY QUANT MY QUANTITATIVE (score 1)
https://quant.stackexchange.com/a/79500
If your strategy is based on intraday data, as in you look at the mean-reversion during the day, then you ignore (remove) the overnight returns. If it's based over several days, then you use end-of-day (eod) data and the jumps are "baked" into the data and won't cause issues with your statistical analysis.
Specifically, if your trading strategy is looking at 5-minute intervals throughout the day and your positions last only 5-minutes, then eod movement has 0 affect on your strategy - so its data can be removed. In this case, you would keep and calculate the returns from 09:30-16:30, but remove the returns from 16:30-9:30 (ignoring out-of-hour trading for simplicity). Thus, you would only have intraday returns.
Generally, there isn't a one size fits all approach to data cleaning. When you remove data, you lose information, but you should understand what information you're losing and how it will affect your results.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.