When to Persist Market Bars and Session Summaries
Summary
The discussion weighs storing precomputed daily, weekly, hourly, or minute bars against resampling them from tick data when needed. Its answer favors avoiding routine aggregate storage when the query system can handle the workload, while highlighting a data-quality problem: trades can arrive late or be corrected after a period closes. Aggregates saved immediately may then need to be recalculated using official exchange records.
Instead, it recommends retaining end-of-session summaries that capture session aggregates alongside order-book state and auction or imbalance statistics. These summaries can reflect exchange-specific sessions and stages, such as opening and closing auctions, regular trading, and post-crossing activity. A daily summary can serve as an end-of-day aggregate, and exchange-published summaries can be compared with calculations from ticks to help find data-quality issues. The advice is experience-based and conditional on query speed and exchange conventions; it does not set a universal persistence threshold or quantify storage and compute tradeoffs.
Key ideas
- Fast query systems can make persisted intraday and calendar-period aggregates less useful.
- Late or corrected trades can make previously saved bars inaccurate and require recalculation.
- End-of-session records can preserve aggregate statistics, order-book state, and auction or imbalance information.
- Session definitions and trading stages vary by exchange.
- Exchange-published session summaries can be used to check aggregates built from tick data.
Tags
Full text
# should I persist the daily/weekly bar data # should I persist the daily/weekly bar data Suppose a small quant group (4 guys) and the tick data is in hand, shall they persist the historical daily/weekly or even smaller scale like hourly/minutes bars data, or just do the on-demand resampling from existing data (minutes from tick, and then resampling to hour/day by minutes)? Could you please advice based on your experience? - if yes/no, please shortly explain: beyond what level bar data should be persisted? - if it depends, please give some best practices. Thanks in advance. ## Answer by Sergei Rodionov (score 1) https://quant.stackexchange.com/a/60987 I find aggregates to be of limited use if the query engine is fast at scale. There is also an issue of out-of-order trades. If aggregates are calculated and persisted right after the period rolls over, you would need to re-calculate them in case missing or corrected trades are present in the official exchange Last Sale log which might be released next day. We recommend to persist end-of-session summaries which include the aggregates for the session, as well as the state of the order book and auction/imbalance statistics. Depending on the exchange you might have two or more sessions, the naming is exchange-specific, for example: - Pre-market - Normal - After-market or - Morning - Day - Evening Further breakdown by stage: - Opening auction call - Opening auction crossing - Normal trading - Closing auction call - Closing auction crossing - Closing auction post-crossing trading In the simplest implementation, a daily Session Summary is equivalent to an End-of-Day statistic, or the Daily aggregate. Some exchanges publish such end of day/session summaries which is useful to compare against aggregates from tick data to find any data quality issues.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.