Handling Variable-Size Market Data in Neural Networks
Summary
The question asks how to represent market inputs whose sizes differ across stocks, such as dividend schedules and volatility surfaces, when training a neural network that expects fixed-size inputs. It notes that principal-component methods can compress yield curves, then asks what alternatives are commonly used for other financial data.
The answer points to padding and recurrent models such as LSTMs as ways to handle variable-length time-series inputs, and links to introductory material on padding, variable-size model inputs, and recurrent forecasting. It does not compare these methods, explain implementation details, or show results on a financial pricing task. Padding therefore appears as a suggested avenue rather than a demonstrated general solution; the links also focus mainly on time series, which may not directly resolve irregular structures such as dividend events or two-dimensional volatility surfaces.
Key ideas
- Padding can make variable-length sequences fit fixed-size model inputs.
- LSTMs are suggested for modeling time series with varying input lengths.
- The response offers external reading links rather than a worked financial example.
- The answer does not establish which representation is best for dividends or volatility surfaces.
Tags
Full text
# Market data and machine learning # Market data and machine learning I have the following general question regarding the use of ML in quantitative finance: Lets say I want to train a model (for simplicity lets consider a neural network), so that I feed some market data for a stock, and I want to compute something on it (the specific goal is not important here, imagine just a pricer). What happens is that usually the market data on a stock has a non-fixed structure. For example for stock $A$ I might have $n$ dividends, whereas for stock $B$ I may have $m$. Therefore, feeding dividend data into the model is not a straight-forward task, as the size of the entry values for the model is not constant and the input for a NN has to be an array of fixed size. I used the dividends example, but this applies to everything: volatility surface (I might have a $n_1 \times n_2$ matrix for one stock and a $m_1 \times m_2$ sized volatility surface for another), etc. For rate curves PCA decomposition is well suited, so it might be simple to transform a variable-sized vector of date-value tuples into a fixed $d$-dimensional structure. But what about other inputs? What is the common practice? I know there are many possibilities in ML, but I would like to know what are the ones that are more readily or well suited to be used in finance. References are also welcome. ## Answer by Mahavir Bhattacharya (score 0, accepted) https://quant.stackexchange.com/a/79079 LSTM and Padding are useful approaches! I'm sharing some links below for your reference. The first one gives a quick glimpse into padding techniques for time series data, using Python (1D CNN). The second and third ones talk about handling data with variable input sizes. The last two dive into the usage of LSTM for time series forecasting. - https://medium.com/full-metal-data-scientist/an-introduction-to-time-series-padding-techniques-in-python-b7307a2eba87 - https://stackoverflow.com/questions/38189070/how-do-i-create-a-variable-length-input-lstm-in-keras - https://datascience.stackexchange.com/questions/48796/how-to-feed-lstm-with-different-input-array-sizes - https://machinelearningmastery.com/lstm-for-time-series-prediction-in-pytorch/ - https://blog.quantinsti.com/rnn-lstm-gru-trading/ Hope this helps! (Disclaimer: I'm associated with Quantinsti, the platform for the 5th link).
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.