Testing Option-Derived Signals in S&P 500 Stock Portfolios
Article Machine Learning for Trading
Summary
This case study examines whether information from listed options can help select S&P 500 stocks. It combines daily equity prices with option-implied volatility levels, skew, term structure, and variance risk premium measures across 633 stocks from 2017 to 2021. Option features are lagged, portfolio choices are made using Friday data, and trades begin at the following Monday open. The research workflow covers return labels, feature construction, several model families, portfolio allocation, risk controls, transaction costs, and a later-period assessment.
The reported validation Sharpe is 2.088, with a 95% interval from 1.005 to 3.117, for a configuration using a linear model, score weighting, ten holdings, and a trailing stop. However, the corresponding model was not evaluated on the 2021 holdout because that period had already informed an earlier modeling lineage. The study therefore treats the reported result as validation evidence only; its out-of-sample effectiveness remains unresolved, and it makes no deployment claim. The stated round-trip cost assumption is 13 basis points, with additional cost stress analysis.
Key ideas
- Lag option-derived predictors to reduce the risk of using information unavailable at decision time.
- Make weekly portfolio decisions after Friday’s close and execute at the following Monday open.
- Compare linear, boosted, deep learning, and latent-factor approaches within walk-forward evaluation.
- Account for trading costs, portfolio construction choices, and predefined risk controls when assessing a signal.
- Treat validation performance as inconclusive about out-of-sample efficacy when the holdout has already influenced earlier research.
Tags
Full text
# Case Study: S&P 500 Equity + Option Analytics
# Case Study: S&P 500 Equity + Option Analytics
This case study trades S&P 500 equities using information from listed options. It combines daily
equity bars with implied-volatility level, skew, term structure, and variance-risk-premium features
for 633 stocks from 2017 through 2021. Decisions use Friday-close information, option features are
lagged one day, and trades execute at the following Monday open. The distinctive question is not
whether options contain information, but whether that information survives point-in-time feature
engineering, weekly portfolio construction, costs, risk controls, and a regime change.
## Dataset Profile
| Property | Value |
|---|---|
| Asset class | S&P 500 equities with option-derived predictors |
| Frequency | Daily inputs; weekly Friday-close decisions |
| History | 2017-2021 |
| Universe | 633 stocks with listed-options coverage |
| Inputs | AlgoSeek S&P 500 daily bars and the materialized daily options surface |
| Primary label | `fwd_ret_5d` |
| Evaluation | Two walk-forward folds; 10-session embargo; 2021 holdout |
| Execution | Friday close to Monday open; one-day lag on option features |
| Cost model | 13 bps round trip at the configured midpoint; 0-50 bps stress grid |
| Current evidence | v3.1 validation carrier: NLinear, score weighted, top 10, 5% trailing stop |
The corrected v3.1 validation Sharpe is 2.088 with a 95% interval of `[1.005, 3.117]`. The
matching NLinear holdout was not run because the 2021 holdout had already been observed on an older
IPCA lineage. The current notebooks therefore present validation evidence, label out-of-sample
efficacy unresolved, and make no deployment claim. The book-aligned v3.0 record remains preserved;
the two result versions are not mixed.
## Pipeline
| Stage | Notebook | Chapter | What it teaches | Writes |
|---|---|---:|---|--------|
| Feasibility | [`01_feasibility_analysis`](01_feasibility_analysis.ipynb) | 6 | Tests options coverage, weekly cadence, and whether equity trading costs leave room for research. | Nothing - the evidence stays in the notebook |
| Labels | [`02_labels`](02_labels.ipynb) | 7 | Builds five- and ten-day return, risk-adjusted, and direction labels with walk-forward boundaries. | `labels/fwd_ret_5d.parquet`, `labels/fwd_ret_10d.parquet`, `labels/fwd_ret_risk_adj_5d.parquet`, `labels/fwd_dir_5d.parquet`, `labels/fwd_dir_10d.parquet` |
| Financial features | [`03_financial_features`](03_financial_features.ipynb) | 8 | Joins lagged IV surfaces to realized volatility, momentum, and liquidity features. | `features/financial.parquet` |
| Temporal features | [`04_model_based_features`](04_model_based_features.ipynb) | 9 | Produces forward-only GJR-GARCH volatility features and documents the pinned single-start feature vintage. | `features/model_based.parquet` |
| Evaluation | [`05_evaluation`](05_evaluation.ipynb) | 7-9 | Audits coverage, staleness, daily IC, and HAC uncertainty before model selection. | `evaluation/triage_ledger.parquet`, `evaluation/ic_timeseries.parquet` |
| Linear models | [`06_linear`](06_linear.ipynb) | 11 | Establishes regularized linear and classification baselines across the label panel. | Training runs and prediction sets in `run_log/registry.db`; coefficients under `run_log/training/{hash}/`, scores under `run_log/predictions/{hash}/` |
| Gradient boosting | [`07_gbm`](07_gbm.ipynb) | 12 | Trains LightGBM configurations and records fold-complete validation predictions. | Training runs and prediction sets; boosters, `learning_curves.parquet`, and `fold_metrics.parquet` under `run_log/training/{hash}/` |
| Tabular deep learning | [`08_tabular_dl`](08_tabular_dl.ipynb) | 12 | Evaluates TabM ensembles on the combined equity and options feature panel. | Training runs and prediction sets; checkpoints under `run_log/training/tabular_dl/` |
| LSTM | [`09_dl_lstm`](09_dl_lstm.ipynb) | 13 | Tests whether recurrent sequence structure improves on point-in-time features. | Training runs and prediction sets; checkpoints under `run_log/training/deep_learning/` |
| PatchTST | [`10_dl_patchtst`](10_dl_patchtst.ipynb) | 13 | Tests multi-scale patch attention and checkpoint stability. | Training runs and prediction sets; checkpoints under `run_log/training/deep_learning/` |
| Latent-factor index | [`11_latent_factors`](11_latent_factors.ipynb) | 14 | Compares the latent-factor family and routes readers to each implementation. | Nothing - it reads the registry |
| PCA | [`11a_pca`](11a_pca.ipynb) | 14 | Fits PCA within each training fold on the persistent panel. | Training runs and prediction sets |
| IPCA | [`11b_ipca`](11b_ipca.ipynb) | 14 | Estimates characteristic-conditioned factors on the ragged panel. | Training runs and prediction sets |
| Conditional autoencoder | [`11c_conditional_autoencoder`](11c_conditional_autoencoder.ipynb) | 14 | Learns nonlinear conditional factor exposures without crossing fold boundaries. | Training runs and prediction sets |
| Stochastic discount factor | [`11d_stochastic_discount_factor`](11d_stochastic_discount_factor.ipynb) | 14 | Estimates a neural SDF and compares checkpoint stability. | Training runs and prediction sets |
| Supervised autoencoder | [`11e_supervised_autoencoder`](11e_supervised_autoencoder.ipynb) | 14 | Learns return-supervised latent factors and reports uncertainty by checkpoint. | Training runs and prediction sets |
| Causal DML | [`12_causal_dml`](12_causal_dml.ipynb) | 15 | Estimates the `ivrv_spread` effect with walk-forward DML and panel-robust inference. | A row in the registry's `causal_runs` |
| Model analysis | [`13_model_analysis`](13_model_analysis.ipynb) | 11-15 | Compares full-coverage families using daily IC with HAC intervals and feature provenance. | Nothing - it reads the registry |
| Equal-weight baseline | [`14_backtest`](14_backtest.ipynb) | 16 | Runs equal-weight top-k baselines and applies coverage-aware selection. | One backtest run per prediction set and entry scheme; `daily_returns.parquet`, `weights.parquet`, `trades.parquet`, `fills.parquet`, `equity.parquet`, `portfolio_state.parquet`, and `spec.json` under `run_log/backtest/{hash}/` |
| Allocation | [`15_portfolio_management`](15_portfolio_management.ipynb) | 17 | Tests five alternative allocators on the ten advancing model configurations. | One backtest run per allocation method, same artifact layout |
| Risk | [`16_risk_management`](16_risk_management.ipynb) | 19 | Compares 14 predeclared fixed controls with paired return uncertainty. | One backtest run per overlay variant, same artifact layout |
| Costs | [`17_costs`](17_costs.ipynb) | 18 | Replays the single best risk-stage configuration across the exact 17-point cost surface. | One backtest run per cost level, same artifact layout |
| Holdout predictions | [`18_holdout_predictions`](18_holdout_predictions.ipynb) | 20 | Refits the selected configuration on all history before 2021 and predicts the holdout. | One training run and one prediction set |
| Holdout backtest | [`19_holdout_backtest`](19_holdout_backtest.ipynb) | 20 | Runs the selected strategy unchanged on the holdout predictions. | One backtest run at `stage='holdout'`, same artifact layout |
| Strategy assessment | [`20_strategy_analysis`](20_strategy_analysis.ipynb) | 20 | Assembles what the case study established, and states which claims the holdout supports. | Nothing - it reads the registry |
## Running
Run the pipeline from the repository root in the locked environment. Set `ML4T_DATA_PATH` to a
directory containing `equities/market/sp500/daily_bars.parquet` and
`equities/market/sp500/options_surface_daily.parquet`. The equity bars ship in this repository at
`data/equities/market/sp500/daily_bars.parquet`, redistributed with AlgoSeek's permission, so no
account, key or license request is needed; cite [algoseek.com](https://algoseek.com) as the source
in anything you publish from them. The options-surface loader provides the materialized research
file when available. Missing data raises an error with acquisition instructions.
```bash
uv sync --frozen
for notebook in \
01_feasibility_analysis 02_labels 03_financial_features 04_model_based_features \
05_evaluation 06_linear 07_gbm 08_tabular_dl 09_dl_lstm 10_dl_patchtst \
11_latent_factors 11a_pca 11b_ipca 11c_conditional_autoencoder \
11d_stochastic_discount_factor 11e_supervised_autoencoder 12_causal_dml \
13_model_analysis 14_backtest 15_portfolio_management 16_risk_management \
17_costs 18_holdout_predictions 19_holdout_backtest 20_strategy_analysis
do
uv run python "case_studies/sp500_equity_option_analytics/${notebook}.py"
done
```
A fresh production run is a several-hour workload on a CUDA-capable machine, with deep learning and
latent-factor training dominating. CPU execution is supported but materially slower. When the
shipped registry and artifacts are present, completed hashes are reused and the notebooks report
each cache hit. Do not skip a model family silently or start downstream from a partial baseline:
the greedy funnel is valid only after all model predictions and all equal-weight baselines exist.
The results source of truth is `run_log/registry.db`, with content-addressed artifacts under
`run_log/training/`, `run_log/predictions/`, and `run_log/backtest/`. Legacy `results/*.json` files
are not used.Shown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.