Scaling Live Trading Features with Historical Data
Summary
This discussion explains how to scale new market features for a predictive trading model after fitting a scaler on historical data. Its central point is to treat scaling as a fitted transformation: estimate its parameters from history, then apply those same parameters to incoming observations. A new value does not need to be part of a multi-row batch; for example, a min-max transform can use historical minimum and maximum values to scale one new observation. That value may fall outside the original zero-to-one range, depending on how the feature changes.
The discussion also stresses matching training and live procedures. During training, each observation should be scaled using only information available before that observation, to avoid lookahead. The history can grow or use a moving window, but that choice affects which past conditions inform the scaling and is an algorithm design decision. The document offers conceptual guidance rather than empirical comparisons of scaling methods; the suitable history policy and the acceptability of out-of-range values depend on the model and application.
Key ideas
- Treat the scaler as a fitted transformation whose parameters come from historical data.
- Apply the historical scaling parameters to each incoming observation, even when it contains only one row.
- A new value can scale outside the range seen during fitting, depending on the transformation and model assumptions.
- Use only prior observations when scaling each training point to reflect live use and avoid lookahead.
- Choose between an expanding history and a moving window based on the intended treatment of older data.
Tags
Full text
# Scaling the data to train, then how to scale the input data?
# Scaling the data to train, then how to scale the input data?
I'm somewhat new into the world of trading algo's, so bare with me. I've made a dataframe with 5 features say. I used preprocessing.scale to scale it. I checked the csv dump of it and it looks fine and dandy. Now, say I've just got todays open, high, low, close, volume data. I've calculated the indicators for it (used a 20 day history to get them). I'd excpect that I would have to scale these new input features so that they'd somewhat match up to what the model was trained with. How can I do that with just 1 row of numbers? Or should I scale the whole 20 day history and hope it works out close to the data scaled for training?
## Answer by madilyn (score 1)
https://quant.stackexchange.com/a/32669
Think of whatever scaling operation that you're using as a model whose parameters were fitted on historical data. The estimated parameters are static.
For example, say one of your features has a minimum value of -3 and maximum value of 5 in your 20 day history, so min-max scaling of 0-1 would take the form:
$\hat{f}: x \mapsto \dfrac{x-x_{min}}{x_{max}-x_{min}},\ x_{min}=-3,\ x_{max}=5$
And you can handle new data points $x$. If you encounter a new feature value of 11 in the future, $\hat{f}\left(11\right)=1.75$. Whether it's valid for your value to fall outside $\left[0,1\right]$ is another issue that depends on your model assumptions.
The other answer before me details the other issue pretty well - how you let the history grow is both a statistical and algorithm design problem. Back to the example I've given, you could let a mix-max scaler scale indefinitely as it requires both $\mathbb{O}\left(1\right)$ memory and amortized $\mathbb{O}\left(1\right)$ time.
## Answer by nbbo2 (score 0)
https://quant.stackexchange.com/a/32099
How do you implement a trading algorithm based on a predictive model? The algorithm is driven by a finite set of past data points, that I will call The History.
Each day you observe new data in the financial markets (for example OHLC for that day). You scale this data by reference to the history that you have (for example you compare today's move to the standard deviation of historical points to conclude that "today is a 2 standard deviation up move"). After this you make a prediction again with reference to historical values. Then the recently observed OHLC is added to the History File.
It is up to you whether you let the history file grow indefinitely (eventually it will have some rather stale data) or whether you drop the oldest data point when you add the newest (sometimes called a moving window approach).
When you trained your algo, you should have done the same thing, i.e. scale each point using only past data points (scale $x_t$ based on $x_{t-1},x_{t-2},\cdots$). Ideally the training should be done similar to the live operation, without lookahead.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.