Choosing Time-Series Windows and Batches for Deep Learning
Summary
The document discusses how to feed time series with uneven observation density into a deep learning model, especially when each batch should retain historical context. It argues that the appropriate representation depends on the model and learning method. Fully connected networks suit observations treated as independent; LSTMs are suggested for autocorrelated sequences and should receive consecutive blocks; ConvNets are suited to multiscale patterns and need batches that preserve their structure.
For window design, the answer considers fixed observation counts versus fixed time durations. It recommends choosing windows long enough to include a useful number of observations and avoid cutting off the patterns the network should learn. Mini-batches should also be composed so the estimated gradient has similar variance across batches. The question’s notion of varying density is acknowledged as unclear, and the advice is general: inspect the data and match batch construction to the model’s assumptions. No comparative experiments or universal best format are provided.
Key ideas
- Input format should match the network architecture and the learning method.
- LSTMs generally need consecutive data blocks, while ConvNets need blocks that preserve expected pattern structure.
- A fixed-duration window can contain different numbers of observations when data density varies.
- Windows should be long enough to retain useful context and avoid breaking expected patterns.
- Batch composition should aim for similar estimated-gradient variance across batches.
Tags
Full text
# time series data modeling for deep learning # time series data modeling for deep learning what is the best format to feed the input data, which are time series with varying density over time, to a deep learning network, while at any iteration we want to feed a batch of data including a historical background? Is it better to consider a constant size of data records or a constant time window including variable data record size? Or is there a better way? ## Answer by lehalle (score 3) https://quant.stackexchange.com/a/48684 I am not sure I perfectly understand your question, the concept of "time series with varying density over time" is not very clear. One thing is for sure, the optimal way to "feed" a neural network is a function of the type of NNet itself and of the learning method you have chosen. For time series - either you believe your data are iid vectors, and you can use a fully connected perceptron - either they are auto-correlated, and probably an LSTM (Long-Short Time Memory) is a good option to consider - if you expect them to have multi-scale patterns ConvNets are probably a good choice. For the learning, the way you feed the NNet has to be compatible with your previous choice: - fully connected perceptrons on iid observations should be trained via Stochastic Gradient Descent (SGD), ie using random small batches - LSTM have to be feed by long blocks of consecutive data, - ConvNets have to be feed by blocks preserving the structure of your expected patterns. Here are few remarks about your last point about using a fixed window in number of observations versus an observation duration: - you need to have a decent number of observations in each batch, hence if you choose a duration, take it long enough - you need to not break the type of pattern you expect the NNet to capture; take your duration (or number of points) long enough. In essence, for mini-batches driving a SGD, their composition has to be chosen such that the estimated gradient (stemming from the averaging over the mini-batch), has the same variance from a mini-batch to the other. Look at your data and make the best choice!
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.