Managing Bar and Tick Data in Parquet Files

Summary

This brief forum post advocates managing market data as files rather than relying solely on a database, particularly when handling large volumes of tick data. The author shares a personal approach using Parquet files and points readers to an implementation associated with the VeighNa ecosystem. The rationale is that file-based storage can be more convenient and efficient for this use case.

The post offers no benchmarks, schema details, query examples, or comparison of storage formats, so its efficiency claim is anecdotal rather than demonstrated. It is a narrow data-engineering note rather than a trading strategy: it does not discuss signal construction, market behavior, or investment results. Readers would need to assess their own access patterns and operational needs before adopting the approach.

Key ideas

• The author recommends file-based management for market data, especially large tick datasets.
• The example uses Parquet as the storage format.
• The post provides no performance benchmarks or implementation details to substantiate its efficiency claim.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.