Building an SQLite Registry for Historical Strategy Optimization Results

Summary

This article describes replacing a growing collection of optimization CSV exports with a portable SQLite registry. The proposed schema stores individual result rows, ingestion records, and deduplicated strategy configurations, while indexes support common searches by instrument, test phase, indicator, parameters, and Sortino ratio. A migration table tracks schema changes as the research pipeline evolves.

A companion export format adds run timestamp, EA version, and unique run identifiers to preserve provenance. The Python ingestion module records source-file hashes to detect duplicate imports, supports automated folder ingestion, and provides queries for comparing historical results, including stability trends and EA versions. The examples show how accumulated records can help establish whether a candidate has passed robustness checks across campaigns, versions, and brokers. The article focuses on research infrastructure rather than trading performance; its usefulness assumes a sequential, primarily single-developer workflow, since SQLite does not support simultaneous writes from multiple processes.

Key ideas

• SQLite can consolidate optimization exports into one portable, queryable research archive.
• Separate tables for results, ingestion events, and strategy configurations preserve both records and their provenance.
• File hashes can help detect duplicate imports that would otherwise distort analyses.
• Run timestamps, EA versions, and unique identifiers support comparisons across research campaigns.
• The design fits sequential local workflows, while concurrent writes are a stated limitation.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.