Align Crypto Price Series Before Testing for Cointegration

# Cointegration between crypto markets


# Cointegration between crypto markets












I'm having an hard time understanding how cointegration works. Basically i'm trying to find cointegrated pairs in the crypto market, so i do the following:

- Get OHLC data for the two markets (i'm getting 5k candles on the 5m timeframe)

- Get the log returns for the two markets

- Check the p-value when i look for cointegration between the log returns

The problem with my code is that i always get very low p-values, no matter what market i try, and on some markets the p-value is 0.

Here is my code:

```
import pandas as pd
import json
import numpy as np
import requests 
import time
import json
import mplfinance as mpf
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import statsmodels.tsa.stattools as ts
import threading
import scipy
import pandas_ta as ta
import ccxt
from pykalman import KalmanFilter

ftx = ccxt.ftx()

def get_ohlc_ccxt(market, timeframe):
    data = ftx.fetch_ohlcv(market, timeframe, limit=5000)
    ohlcv = pd.DataFrame(data, columns=['time', 'open', 'high', 'low', 'close', 'volume'])
    ohlcv = ohlcv.drop_duplicates(subset=['time', 'open', 'high', 'low', 'close', 'volume'], keep='first')
    ohlcv['time'] = ohlcv['time'].astype('int64')
    ohlcv['time'] = ohlcv['time']/1000
    ohlcv['date'] = pd.to_datetime(ohlcv['time'], unit='s')
    ohlcv = ohlcv.set_index(pd.DatetimeIndex(ohlcv['date']))
    return ohlcv

def check_pair(first_market, second_market, timeframe):
    first = get_ohlc_ccxt(first_market, timeframe)
    second = get_ohlc_ccxt(second_market, timeframe)

    if len(first) != len(second):
        length = min(len(first), len(second))
        first = first.iloc[-length:]
        second = second.iloc[-length:]

    x = first['close'].to_numpy()
    y = second['close'].to_numpy()
    
    first['logret'] = first.ta.log_return()
    second['logret'] = second.ta.log_return()

    xr = first['logret'].fillna(0)
    yr = second['logret'].fillna(0)

    coint = ts.coint(xr.to_numpy(), yr.to_numpy())
    p_value = coint[1]

    print('Cointegration:', first_market, second_market, p_value)
    spread = xr-yr
    
    return first, second

data = check_pair('BTC-PERP', 'ETH-PERP', '5m')
```

In this case the p-value is `8.94059749424772e-29`. If i try other markets (like BTC-LINK, or BTC-LTC and so on) the p-value will always be incredibly low. Can anyone help me find what i'm doing wrong?

## Answer by Hamish Gibson (score 2)

https://quant.stackexchange.com/a/71495

There is likely an issue with the missing candles in your data. Whilst you have made steps to ensure that the data is equal length for your test, it is quite possible that these two series' timestamps do not align and therefore, ultimately end up covering different time periods- the data is not in sync.

Allow me to provide an example, suppose you look at the `BTC-USD` and `ETH-USD` markets. You have already set a limit of 5,000 candles at 5-minute intervals to be returned, you provide all the parameters to the CCXT library, and it returns you your time series. Let me now denote the `BTC-USD` time series as $X_{t}$ and the `ETH-USD` series as $Y_{t}$, where for each series we have $t$ such that $0 \leq t < 5000$.

5,000 5-minute intervals take us back just under 2.5 weeks. So we have our two series such that $X_{0}$ and $Y_{0}$ fall on, let's say 20th June 2022 00:00:00.

However, for whatever reason, let's say that for `ETH-USD`, there were some number of periods where no trades took place. This can be due to, illiquid markets on illiquid exchanges, exchange outages, a faulty API etc... The point is, it is entirely possible no trades took place during certain 5-minute interval(s). I can see you have used FTX as your example, I cannot comment on their API documentation but I do know that is common for other exchange APIs to not return anything for candle intervals where 0 trades took place. See this example about the Bitfinex API, which returns no timestamp when there is no candle data due to no trades occurring.

This can cause your series to be less than 5,000 intervals in length. Whilst your method is correct in ensuring that the two time series are of equal length, you need to make sure it is of equal length in the right places. Your code arbitrarily truncates the series, but what you need to do is only use data where timestamps exist in both series!

Reverting back to my example, let's say that `ETH-USD` had 3 missing intervals / candles. Let's say that at the $x$, $y$, and $z$ 5-minute interval where ($x,y,z \leq 5000$), no trades occurred. But your `BTC-USD` series is whole and complete, then you can only use data $X_{t, \text{s.t.} 0 \leq t < 5000, t \notin \{x,y,z\}}$. This ensures your series are correctly aligned and your cointegration tests are being correctly done.

## Answer by Dave (score 1)

https://quant.stackexchange.com/a/71520

This is a common observation when hypothesis tests are performed on large data sets.

Most common hypothesis tests have the property that, as sample sizes increase, the test gets increasingly sensitive, in the limit being perfectly sensitive to any deviation from the null hypothesis.

Consequently, a large sample size gives your test tremendous power to reject even tiny deviations from the null hypothesis. I suspect this is what has happened to you: there is at least a tiny deviation, and the large sample size allows you to detect it.

(Note that this sensitivity occurs when the null hypothesis truly is false. If the null hypothesis is true, then the sensitivity should be around $\alpha$, meaning a $\text{Uniform}(0, 1)$ distribution of p-values under the null hypothesis.)