Building a Native MQL5 Kafka Producer for Trading Signal Fan-Out
Summary
This article describes how an MQL5 Expert Advisor can publish structured trading signals to Apache Kafka using raw TCP sockets and a hand-built implementation of the Kafka producer protocol. It covers big-endian fields, base-128 varints with zigzag encoding, CRC32C checksums, Produce request framing, and Kafka RecordBatch v2 encoding. A two-pass process patches batch length and checksum fields after records have been assembled. The signal payload also carries a schema version, with a startup check intended to catch mismatches between the declared and serialized fields.
The design sends each signal to a topic so the broker can distribute it to independent consumers and retain records for later replay. The article describes retry and latency metrics and proposes validating encoded records with a separate consumer script. Its scope is producer-side only; consumer-group coordination and a production consumer are left for separate work. The hand-built protocol, socket handling, schema agreement, and broker configuration create integration and operational concerns. The article explains an architecture and implementation, but does not establish trading performance or delivery guarantees under all failures.
Key ideas
- A Kafka topic lets one MT5 producer publish signals for multiple independent consumers to read and replay.
- A native MQL5 producer must implement protocol details including framing, varints, CRC32C, and RecordBatch v2 encoding.
- Batch length and checksum fields can be patched after the records are encoded.
- A schema version and field-count check can help detect payload contract mismatches before the EA starts.
- The article covers publishing and validation, while consumer coordination remains outside its scope.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.