Testing Alternative Data Signals with Preregistration and Robust Validation

# Framework for hypothesis testing and non-linear signal extraction in alternative data


# Framework for hypothesis testing and non-linear signal extraction in alternative data












I am looking to understand the practitioner research framework for evaluating alternative datasets (e-receipts, healthcare claims, airline bookings). Because many of these datasets are now widely commoditized, standard linear relationships are often priced in. I am specifically interested in the statistical methodology used to rigorously test for alpha and capture second-order effects. ​

Hypothesis formulation: What is the standard process for formulating a specific, testable hypothesis for an alternative dataset, and what criteria are used to falsify it?

​Effect Isolation: Once a hypothesis is formed, how do you formally test and isolate that specific effect to ensure you are measuring the actual data signal and not some noise?

​Non-Linear Techniques: Given the decay of simple linear signals in commoditized datasets, what is the accepted methodology for applying non-linear models (tree-based methods/boosting) to capture second-order effects without curve-fitting?

​I am interested purely in the structural research methodology and statistical frameworks, not proprietary signals. Any reference to literature would be highly appreciated.

## Answer by Adol (score 0)

https://quant.stackexchange.com/a/85742

I will answer from a running pre-registration process rather than from the literature, since you asked for practitioner methodology.

Hypothesis formulation and falsification. Write the gate before you look, in a document that is committed and dated. The specific rule that changed our outcomes: cite the minimum detectable effect of the estimator you are about to use, and do not set a gate line inside it. We adopted that after discovering we had been gating on our weakest instrument. The MDE was 22.3 percentage points at six day clusters, and the gate we had written was far tighter than that, which means the test could not have distinguished a pass from a failure no matter what the data did. A gate inside your MDE is theatre, and it is very easy to write by accident because the number looks rigorous.

Effect isolation. Two things, in order. Residualize against the obvious factor before you believe any correlation. An alt data series that tracks attention will mostly reproduce market beta, and the raw IC will look encouraging for that reason alone. Our residual IC against a market factor is indistinguishable from zero, and that is the honest headline for the raw signal. Then cluster your errors by day. Observations from the same day are not independent, and if you treat them as independent your effective n is inflated by roughly the average number of observations per day, which in our case was an order of magnitude.

Non-linear without curve fitting. The guard that earns its keep more than any model choice is an equal budget random search control. Run your TPE or Bayesian study, then run plain random search with the same trial budget on the same folds. If they tie, the surface is flat and your boosted model has found nothing that the search procedure did not hand it. That is a finding, and it is publishable internally, not a failed experiment. Around that:

- Select on Wilson lower bounds rather than point estimates, so a variant cannot win by shrinking n. This kills a whole family of apparent improvements that are just fewer trades.

- Penalize cross fold standard deviation in the objective. You want stability, not a fold that got lucky.

- Embargo gaps between folds sized to your maximum label horizon, or your folds leak into each other through the labels rather than through the features.

- One final holdout, physically withheld from the optimizer, queried exactly once, and never used to reorder the shortlist.

- State the multiple testing count explicitly in the writeup. Not a correction necessarily, but the count, so the reader can apply their own.

The uncomfortable part is that most of what this process touches dies. We have killed the same feature family twice under pre-committed gates, the second time on a fresh holdout after it looked promising again. That is the framework working. If your framework is not killing most of what you feed it on commoditized data, the gates are probably inside the MDE.