构建与评估合成金融数据

文章《交易机器学习》

总结

本章概述生成合成金融数据的方法，以及如何判断生成结果是否有用。内容从已实现市场路径数量有限这一事实出发，并讨论反复搜索策略可能抬高表面回测表现的风险。章节比较了可解释的模拟基线方法，包括自助法和随机模型，以及用于结构化表格的学习型生成器，例如 GAN、扩散模型和语言模型。示例涵盖收益序列、尾部风险、时间结构、不规则观测和混合类型金融数据。

评估围绕保真度、下游效用和隐私展开，这些目标可能相互冲突。诊断方法包括风格化事实、依赖关系、条件行为、基于任务的比较（例如使用合成数据训练、使用真实数据测试）以及隐私检查。本章的核心提醒是：看似合理的样本可能无法反映罕见事件或依赖关系变化；学习型模型可能泄露信息、放大偏差、过拟合训练数据，或生成的新情景新颖性有限。合成路径可以拓宽稳健性分析，但不能取代真实市场证据，也不能免除谨慎验证的必要性。

核心观点

在有限的历史记录上反复自适应地搜索策略，可能夸大回测表现。
自助法和随机模拟方法为学习型生成器提供可解释的基准。
生成器的选择应符合数据结构和下游任务。
应从保真度、效用和隐私方面评估合成数据，因为这些目标可能相互权衡。
看似合理的样本仍可能无法复现尾部事件、依赖关系变化或条件动态。

# ETF Universe (Yahoo Finance)


# ETF Universe (Yahoo Finance)

100 diversified ETFs spanning nine thematic categories — the red-thread
universe for the Ch6 momentum strategy and the ETF case study that runs
through Ch11-20. Daily OHLCV back to 2006.

## Dataset

- **Source**: Yahoo Finance, pulled via the `yfinance` Python package.
- **Coverage**: 2006-01-01 → present, daily OHLCV.
- **Symbols**: 100 ETFs across 9 categories.
- **Size on disk**: ~29 MB.
- **Runtime**: ~2-3 minutes for a full refresh (yfinance uses public
  Yahoo Finance endpoints with light rate-limiting).
- **API key**: not required.
- **License / attribution**: Yahoo Finance data is free for personal
  and educational use (https://policies.yahoo.com/us/en/yahoo/terms/index.htm).
  Redistribution of the raw OHLCV is not permitted; derived analytics
  (returns, features, model outputs) are fine. When publishing results,
  cite Yahoo Finance as the source.

## Categories

| Group                    | Count | Symbols                                                                                          |
| ------------------------ | ----- | ------------------------------------------------------------------------------------------------ |
| US Equity — Broad        | 10    | SPY, QQQ, IWM, DIA, VTI, MDY, IJR, RSP, IVW, IVE                                                 |
| US Equity — Style        | 10    | VTV, VUG, MTUM, QUAL, VLUE, USMV, DVY, SDY, VIG, SCHD                                            |
| US Sectors               | 13    | XLB, XLC, XLE, XLF, XLI, XLK, XLP, XLU, XLV, XLY, XLRE, VNQ, IYR                                 |
| International Developed  | 18    | EFA, VEA, VGK, IEFA, ACWI, ACWX, EWJ, EWG, EWU, EWT, EWH, EWQ, EWL, EWN, EWI, EWP, EWC, EWA      |
| Emerging Markets         | 11    | EEM, VWO, IEMG, FXI, MCHI, EWZ, EWY, EWW, INDA, EZA, THD                                         |
| Fixed Income             | 15    | AGG, BND, BNDX, TLT, IEF, SHY, GOVT, BIL, LQD, VCSH, HYG, JNK, TIP, EMB, MUB                     |
| Commodities              | 9     | GLD, IAU, SLV, PPLT, USO, UNG, DBC, GSG, DBA                                                     |
| Specialty                | 10    | IBB, XBI, SMH, SOXX, KRE, XME, OIH, XRT, ITB, ITA                                                |
| Currency                 | 4     | UUP, FXE, FXY, FXB                                                                               |

Full symbol list + category tagging: `config.yaml`.

## Download

```bash
uv run python data/etfs/market/download.py              # all 100 ETFs
uv run python data/etfs/market/download.py --symbol SPY # single symbol
uv run python data/etfs/market/download.py --dry-run    # plan only
```

Output layout under `$ML4T_DATA_PATH/etfs/market/`:

```
etf_universe.parquet                    # consolidated 100-ETF daily OHLCV (loader target)
ohlcv_1d/ticker=<TICKER>/...            # hive-partitioned per-symbol bars (provider-native)
etf_universe_dictionary.parquet         # symbol metadata (symbol, group, description)
etf_universe_metadata.json              # dataset metadata (row counts, date span)
```

## Loading

```python
from data import load_etfs

df = load_etfs()                                             # all 100 ETFs
df = load_etfs(symbols=["SPY", "QQQ", "IWM"])
df = load_etfs(start_date="2020-01-01", end_date="2024-12-31")
```

Schema (canonical):

| Column      | Type     | Description   |
| ----------- | -------- | ------------- |
| `symbol`    | String   | ETF ticker    |
| `timestamp` | Date     | Trading date  |
| `open`      | Float    | Opening price |
| `high`      | Float    | High price    |
| `low`       | Float    | Low price     |
| `close`     | Float    | Closing price |
| `volume`    | Int      | Trading volume|

## Consumers

- **Ch2**: `01_etf_eda.py`.
- **Ch6**: `01_etfs_setup.py` (strategy definition).
- **Ch8**: `01_price_volume_features.py`, `03_structural_cross_instrument_features.py`, `05_feature_selection.py`, `06_robustness_sensitivity.py`, `07_event_studies.py`.
- **Ch16**: `06_framework_parity.py`, `09_performance_reporting.py`, `11_sharpe_ratio_inference.py`.
- **Ch18**: `01_cost_taxonomy.py`, `03_market_impact_calibration.py`, `06_ml4t_execution_demo.py`.
- **`case_studies/etfs/`**: full pipeline from `01_feasibility_analysis.py` through
  `20_strategy_analysis.py` — the flagship reader case study.

在遵守原作品许可的前提下，附作者信息全文展示。许可协议: MIT

此摘要由 Stratmill 研究智能体根据原文撰写，并非原文副本。

构建与评估合成金融数据

总结

核心观点

标签