Jacobian Rank Limits in Decision-Focused Learning

Summary

This study analyzes how predictor geometry constrains decision-focused learning, where model training is tied to a downstream decision objective. Using sparse index tracking, it distinguishes the covariance information used by an optimizer from the parameter-update directions available to the predictor. Rank-one predictor Jacobians yield collinear nonzero per-example gradients, while a spectral bound describes near-collinearity. The paper also characterizes batch update subspaces and gives counterexamples showing that local rank properties alone do not determine shared minimizers or collinear batch updates.

Experiments test whether these geometric limits affect decision quality. In the reported equity configurations, decision-focused learning improves little over mean squared error; other experiments find larger regret reductions for shortest-path and knapsack tasks, with the corrected result persisting only for knapsack. Capacity, coordinate scaling, and training setup affect optimization outcomes. Financial forward-target controls and a matched neural comparison show no aggregate decision-focused advantage in the tested architecture. The findings are specific to the examined models and tasks: Jacobian structure explains available learning directions, but held-out decision quality is needed to establish practical value.

Key ideas

• A predictor's Jacobian describes which parameter-update directions are available to decision-focused learning.
• Rank-one Jacobians make nonzero per-example gradients collinear, while batch updates need not share that property.
• Local rank constraints do not by themselves imply common minimizers.
• Experimental gains vary across tasks, with corrected evidence persisting only for knapsack among the cited comparisons.
• Coordinate scaling changes optimization behavior, so held-out decision quality remains essential for evaluation.

Tags

# Jacobian Rank Collapse in Decision-Focused Learning


# Jacobian Rank Collapse in Decision-Focused Learning









Decision-focused learning (DFL) trains predictors through downstream objectives, but a different loss need not provide an independent parameter-update direction. We characterize this restriction through the predictor Jacobian, using sparse index tracking to distinguish the covariance entries read by the optimizer from the parameter directions available to learning. Rank-one Jacobians make nonzero per-example gradients collinear; a conditional spectral bound describes near-collinearity. A batch-subspace characterization and counterexamples show why these local statements imply neither common minimizers nor collinear batch updates. Experiments examine when geometry translates into decision quality. Across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%; a 385-parameter conditional predictor also has pointwise rank one. In validation-tuned shortest-path and knapsack experiments, full-capacity SPO+ reduces mean regret by 11.6% and 10.6%, respectively; only knapsack survives correction across eight comparisons. The capacity contrast persists on fresh datasets across batch orders and training budgets. Holding expressivity fixed, invertible coordinate scaling lowers spectral effective rank and ordinary SGD gains; compensating for the scaling restores the original trajectories. Financial forward-target controls separate forecast accuracy from decision quality; a matched neural comparison finds no aggregate DFL advantage in the tested architecture. These findings distinguish local rank restrictions, coordinate-dependent optimization and predictive accuracy. Predictor geometry helps explain available learning directions, while held-out decision quality remains the test of practical benefit.

Shown in full with attribution under the source's licence. Licence: abstract CC0

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.