Two-Phase Commit as a Coordination Pattern for Distributed Backtests
Summary
This short discussion considers how to coordinate components of a distributed trading system during backtesting when a market tick no longer provides a sufficient synchronization point. It proposes borrowing the two-phase commit pattern from database transaction processing: a coordinator issues a uniquely identified request, participants acknowledge readiness, and the operation is accepted only after the required confirmations. Incomplete operations can be recognized and retried.
The answer frames this as a conceptual architecture rather than a tested backtesting design. It notes that long chains of components require coordination to be represented in the protocol, and that message formats and transport choices affect speed and resilience to interruption. No implementation, benchmark, or trading result is supplied, so the pattern would need adaptation to the simulator's event ordering, state recovery, and reproducibility requirements.
Key ideas
- A market tick may not be a sufficient synchronization marker when simulated components interact in feedback loops.
- Two-phase commit offers a pattern for coordinating multi-machine operations through request, acknowledgment, and acceptance stages.
- Unique request identifiers can help distinguish transactions and support retries after incomplete exchanges.
- Long interaction chains and transport reliability need to be reflected in the design.
Tags
Full text
# Need advice about distributed backtesting architecture # Need advice about distributed backtesting architecture We are working under complex enough distributed trading system where several components will run on different physical machines. Unfortunately, I'm stuck on part backtesting part. Originally we was planning to use tick as synchronization marker. Idea was working till we not added more complicated interaction logic when components started to interact with each other with back loop. I'm sure that this problem was solved many times before and dont want to reinvent the wheel... Can anybody share at least basic information about topic? ## Answer by Allen Maxwell (score 0) https://quant.stackexchange.com/a/25780 In database-speak, this problem has been solved with things like two-phased commit. That's used in transaction (money) processing when the machines are separate and nothing can be lost. That design required a unique token (like your Tick would have) but generic (not a tick but your own creation). The process is one machine says "I'm ready to do 'A' if you are." Machine B responds with an acknowledgment on that request token. Then an acceptance from A to B defines a Complete. If no answer from A or B along the way, the transaction is incomplete. A retry can happen. The medium for making and sending tokens/requests is up to you. This is the conceptual design and helped make Oracle a fortune 500 company. With JSON and various other post-SOAP era (REST and all the variations) web-service contracts, you can build a good infrastructure. The development language and tools will make it easy or hard for you. I made an asset/machine-tracking system (with Satellite and Cellular data) all built on similar concept. It supported REST and other formats for sending tokens (from my server to a satellite that talks to a fire-fighting pump). If it's a long chain, you need to build that into your design. You need to know what formats are super fast and survive interruptions at the proper layer: eg TCP vs UDP. Sounds like a fun project.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.