Modeling High-Frequency Price Autocorrelation Across Time Scales

# How to model financial HFT time-series data with multi scale autocorrelation


# How to model financial HFT time-series data with multi scale autocorrelation












I work with tick level time-series univariate prices data. Tick level means that there are hundreds to thousands observations per second. The observations are timestamped, so one can use both wall clock and event time (or whatever you want). Most of the price changes are very small (1e-5 of %), but the small changes add up and there might be occasional big price changes. Sometimes there is little activity, sometimes there are clusters of high activity (bigger and more frequent price changes). There is strong autocorrelation structure at the scale of milliseconds and seconds due to market microstructure. Strong means that the difference between current price and the predicted price is big enough to make money in practice. There is also weaker autocorrelation at longer time scales (minutes, hours, days), but as the effects are more long-lasting, they add up and it is easier to exploit it because you have more time to react, so actually these long-term autocorrelations are also important in practice.

The goal is to model the autocorrelation structure of the price process to get a long-term price forecast that would incorporate the information about autocorrelation at different scales. I am looking for a solution that works at least somehow:

- it is computationally feasible,

- it doesn’t leave out huge (like 50%) chunks of information

The solution doesn’t have to be grounded in formal theory (specifying a proper stochastic process and estimating its parameters) and it doesn’t have to be perfect (a consistent, or even “almost kind of consistent” estimator is enough, no efficiency or unbiasedness is needed). In my opinion, please prove me wrong, for the datasets of this size (1bln univariate observations), computational feasibility means OLS, probably with online/chunk updates.

What are the options known to me?

- Self-exciting point processes. This is theoretically the right way to model this situation. However, to the best of my knowledge (I’ve never seriously tried them), it is computationally feasible only for an unrealistically simplified case that loses most of the information (100k observations subsample, a simple kernel that captures only positive autocorrelation at one scale). Does anybody know if these are successfully used in practice in finance?

- ARIMA. Just work in event time and you have equally spaced observations. MA is not very useful, because of the empirical autocorrelation structure and computational complexity. So ARIMA actually means AR. A big advantage is that you can get an integrated n-step ahead forecast (sum of expected future price changes n steps ahead) in a closed form. The problem here is that if you want to capture different autocorrelation at both tick frequency and 1 second frequency, then with 1000 observations per second you need AR(1000). A simple AR(1000) would be both computationally infeasible and would give way too much “preventable” variance in the estimates. And for 1 tick and 1 day frequency one would need AR(86 400 000). I think that this is a solvable problem using a HAR (Hererogeneous AR) approach, some kind of sampling and iterative aggregation at different frequencies (even if this would be totally unsound from a theoretical perspective).

- ARFIMA and other fractal and multi scale stuff. Sound relevant, but computationally infeasible with no gain to show for it. I’ve tried it many times and it totally doesn’t work. Does anybody know if these are successfully used in practice in finance?

- Frequency domain. I was looking for non/semiparametric estimators for the auto-correlation function. Working in frequency domain could also be used to diagnose how much variance is there on different frequencies. I didn't get any encouraging results here, maybe it is totally irrelevant, maybe I am just very weak in this area.

- Continuous time task formulation. I don’t even know how to formulate a momentum (positive returns autocorrelation) continuous time process. I think this might help with conceptual understanding, but I don’t see how this helps with empirical estimation.

- Subsampling. Use various smart subsampling schemes to reduce the amount of data while retaining most of the information. Event time, cusum on price change, cumulative realised volatility. I think this must totally be part of the solution.

I see the biggest opportunity in the following approach. Firstly, use an unusually large “regularised” (or semiparametric?) AR model (~20 parameters, taking into account lags up to ~1000), for example something like HAR (Heterogeneous AR). This still doesn’t allow to combine daily and tick frequency in one model. So, secondly, we need to use one model to estimate high frequency component, then get its predictions, subsample data to decrease frequency, estimate a model for a different frequency. Such multi scale model combination sounds totally theoretically unfounded (a subsampled AR process is not AR) and I am not sure if it would work in practice.

Do you know

- a solution

- a potential approach that I’ve missed here

- what is used in practice by quants?

- non/semiparametric approaches to estimating the auto-correlation function

This might be a bit of an open-ended and (hopefully) discussion-inspiring question, but it is very practical for me and I think it would be very practically useful for other quant traders.

Crossposted from https://stats.stackexchange.com/questions/497533/how-to-model-financial-hft-time-series-data-with-multi-scale-autocorrelation?noredirect=1#comment920679_497533