ETC–DASH Cost-Aware Linear Actor–Critic Pair Trading with Five Exposure Levels and Causal Rolling Retraining (Binance USD-M 4H)

Derived by the Research Lead from Yang and Malik, 'Reinforcement Learning Pair Trading: A Dynamic Scaling Approach': we operationalize their RL2 idea (an actor-critic policy that scales signed pair exposure instead of trading a fixed on/off z-score rule). This is NOT a reproduction of their BTCGBP/BTCEUR experiment or its reported performance. Every rule below is our choice and is stated completely; the developer needs nothing outside this text. INSTRUMENTS AND BARS. Primary ETCUSDT.BINANCE (USD-M linear perp, 4-HOUR bars), secondary DASHUSDT.BINANCE (4-HOUR bars, passed as extra_instruments / extra_bar_types). Keep the latest close of each leg; run the decision step only when both legs have a bar with the same close timestamp (synchronized bar t). If one leg has no bar at t, skip t entirely (no feature update, no action). Leverage config 1.0. STEP 1 - SPREAD AND Z-SCORE (each synchronized bar t, W = spread_window, baseline 360 bars = 60 days). x_i = ln(DASH close), y_i = ln(ETC close) over the last W synchronized bars INCLUDING t. OLS y = alpha + beta*x on those W points. Residuals e_i = y_i - alpha - beta*x_i; sigma_t = population std of the W residuals; z_t = e_t / sigma_t, clipped to [-4, 4]. If beta <= 0.2 or beta > 5 the pair is 'unhedgeable' at t: all actions except exposure 0 are masked (an open position is closed). Leg weights for sizing: wE = 1/(1+beta), wD = beta/(1+beta) (gross weights sum to 1). Keep a history of z_t, beta_t, sigma_t, wE_t, wD_t and the log closes for training. STEP 2 - OBSERVATION VECTOR (8 features, all computed from data at or before t; e_prev = current signed exposure level before acting, hold = synchronized bars since the current position was opened, 0 if flat): f1 = 1 (bias) f2 = z_t / 2 f3 = clip(z_t - z_{t-1}, -2, 2) f4 = clip(z_t - z_{t-6}, -4, 4) / 2 f5 = e_prev f6 = e_prev * z_t / 2 f7 = min(hold / max_hold_bars, 1) f8 = clip(std(d_s over last 30 bars) / std(d_s over last W bars) - 1, -1, 2), where d_s_i = (y_i - y_{i-1}) - beta_t*(x_i - x_{i-1}) using the current beta_t. STEP 3 - ACTIONS. Five discrete exposure levels A = {-1.0, -0.5, 0.0, +0.5, +1.0}. Level e means signed pair exposure: e > 0 = LONG the spread = long ETC with notional |e|*G*wE*equity and short DASH with notional |e|*G*wD*equity; e < 0 = short ETC / long DASH with the same notionals. G = gross_exposure fixed 1.0 (total gross notional at |e| = 1 is 1.0x equity; the largest leg is below 0.85x equity). equity = get_account_equity(). STEP 4 - ACTION MASK (applied identically in training and in trading). Start with all five allowed, then: (a) Direction: if z_t > 0 mask e > 0 (do not buy the rich leg); if z_t < 0 mask e < 0. (b) Opening/increasing zone: any level with |e| > |e_prev| (opening or scaling up) is masked unless |z_t| >= open_z (baseline 1.8). (c) Cost hurdle: any level with |e| > |e_prev| is also masked unless expected capture (|z_t| - close_z) * sigma_t * 100 >= min_capture_pct (baseline 0.6 %, about 3x the 0.20 % per-leg-notional pair round trip). (d) Forced close: if |z_t| <= close_z (baseline 0.4), or the sign of z_t is opposite to the sign of e_prev (spread crossed through zero), or hold >= max_hold_bars (baseline 60 = 10 days), or the pair is unhedgeable, the ONLY allowed level is 0. (e) Re-entry block: after a forced close by max_hold_bars or by the pair stop (Step 8), opening in the same direction is masked until |z_t| has once fallen below open_z. Level 0 is always allowed, so the mask is never empty. Scaling down (|e| < |e_prev|, same sign) is always allowed unless (d) forces 0. STEP 5 - ACTOR AND CRITIC (linear, our choice of 'small'). Actor: theta is a 5x8 matrix; logits_k = theta_k . f; masked softmax pi_k = exp(logit_k)/sum over allowed actions (0 for masked). Critic: w is an 8-vector; V(f) = w . f. STEP 6 - REWARD (net mark-to-market, per synchronized step, in percent of equity). For the transition t -> t+1 with chosen level e_t and previous level e_prev: R = wE_t*(y_{t+1} - y_t) - wD_t*(x_{t+1} - x_t) (log spread return of a unit long-spread position, weights frozen at t); reward = 100 * [ G*e_t*R - c*G*|e_t - e_prev| ], with c = cost_per_side fixed 0.0008 (0.05 % taker + ~0.03 % slippage per unit gross notional traded). Funding is not in the reward (the backtest charges it in PnL). STEP 7 - TRAINING (causal, from scratch, then frozen). Trading is flat until N_sync >= spread_window + training_bars + 30 synchronized bars have been seen. At that bar and then every retrain_bars = 180 synchronized bars (fixed, 30 days) retrain: take the last training_bars (baseline 1620 = 270 days) completed transitions, i.e. bars ending at t-1 -> t (only data already observed, never future bars). Use the stored z, beta, sigma, weights and log closes for those bars (each computed causally at its own bar, so the spread_window bars before the slice are required warm-up). Reset theta = 0 and w = 0. Run n_epochs = 30 (fixed) passes; each pass walks the slice in time order starting from e_prev = 0, hold = 0, simulating the agent's own position (features f5-f7 and the mask use the simulated position; forced-close and re-entry rules apply; the pair stop is not simulated). At each step: build f and mask, sample action a from pi using numpy.random.default_rng(seed = 12345 + retrain_index) (one generator per retrain, deterministic), compute reward r and the next features f' (with the simulated new position); at the last step of the slice use V(f') = 0. TD error delta = clip(r + gamma*V(f') - V(f), -5, 5) with gamma baseline 0.9. Updates: w <- w + 2*learning_rate*delta*f; for every allowed k: theta_k <- theta_k + learning_rate*delta*(1[k=a] - pi_k)*f (masked actions unchanged). learning_rate baseline 0.01. After training, theta and w are FROZEN until the next scheduled retrain; trading decisions never update them. Training happens only inside the strategy on bars already received, so no validation or holdout bar is ever used to fit the policy; when the platform restarts a window with warm-up bars, the strategy rebuilds history from those warm-up bars and trains at its first eligible bar exactly as above. STEP 8 - LIVE DECISION AND EXECUTION. At each synchronized bar after the first training: compute f and the mask from the REAL position, choose a* = argmax over allowed actions of pi (greedy; ties -> the level with smaller |e|, then 0). If a* != e_prev, rebalance both legs to the target notionals of a* using the current wE_t, wD_t (size = notional / close, rounded to instrument precision); if a* == e_prev, do nothing (no rebalancing for beta drift). Orders are market orders submitted at the synchronized bar's close: on this continuous 24/7 venue this is the platform's registered equivalent of next-bar-open execution. If after fills the legs are unbalanced (self.legs_balanced() false, e.g. a refused order), close both legs and set e = 0. Pair stop (catastrophic cap): if the combined unrealized PnL of the two open legs falls to -pair_stop_pct (fixed 4 %) of equity, checked on each synchronized close, close both legs at market and apply re-entry block (e). On any close, reset hold to 0. Track hold from the time the position went from 0 to non-zero (scaling does not reset it). RISK CONTROL route (b): signal exits (z reversion/crossing, policy action, 10-day time stop) with capped sizing (gross 1.0x equity, no leg above 100 % of equity), plus the 4 % pair stop as a disaster cap. Intended strategy volatility ~15-20 % annualized. VENUE CHANGE vs the registered version: execution moved from Hyperliquid to Binance USD-M because Hyperliquid DASH 4H bars start 2025-08-04 (about 2,600 synchronized bars), which the warm-up plus training slice alone would almost consume. FEE ARITHMETIC: an entry at |z| ~1.8-2.2 closed at |z| <= 0.4 captures ~1.4 sigma of the log residual; with a 60-day residual sigma typically 4-8 % for this pair, a realistic partial capture of ~0.5-1.0 % of gross notional per round trip against ~0.10 % taker cost per unit gross (plus slippage, ~0.16 %) leaves roughly 2-5x the cost; the min_capture_pct mask enforces >= 0.6 % expected capture before any exposure increase.