본문으로 건너뛰기
라이브러리 문서 전체

호가창 틱 데이터: MBO 및 NASDAQ ITCH 데이터 소스

노트북 Machine Learning for Trading

요약

이 가이드는 미시구조 연구에 필요한 US 주식 호가창 틱 데이터의 두 가지 출처를 소개합니다. Databento의 주문별 이벤트와 NASDAQ TotalView-ITCH 메시지입니다. 형식, 커버리지 특성, 대략적인 데이터 크기와 장단점을 설명합니다. MBO는 추가, 변경, 취소, 체결과 같은 처리된 주문 이벤트를 제공하고, ITCH는 파싱과 정규화가 필요한 무료 원시 바이너리 피드입니다.

워크플로는 MBO 데이터를 받기 전에 다운로드 비용을 추정하고, 선택한 날짜의 데이터를 내려받고, 이용 가능한 데이터를 불러와 이벤트나 메시지 유형을 살펴보는 과정을 다룹니다. 유료 MBO 요청은 필요한 종목과 기간으로 제한하고, ITCH는 저장 공간과 파싱 작업이 많이 드는 대신 더 폭넓은 분석에 활용할 수 있다고 권합니다. 이 자료는 트레이딩 전략이나 실증 연구가 아니라 데이터 접근 및 탐색 가이드이며, 호가창 분석 결과를 제시하지 않습니다.

핵심 아이디어

  • MBO와 ITCH는 호가창 재구성과 미시구조 연구에 적합한 주문 단위 이벤트 데이터를 제공합니다.
  • MBO는 종목과 날짜별로 처리되어 유료 제공되고, ITCH는 파싱이 필요한 무료 원시 데이터입니다.
  • MBO 데이터를 요청하기 전에 비용을 추정해야 합니다.
  • ITCH 파일은 분석 전에 상당한 저장 공간과 타임스탬프 정규화 작업이 필요합니다.
  • 이 가이드는 데이터 워크플로를 설명하지만 트레이딩 전략을 평가하지는 않습니다.

태그

주식시장 미시구조주문 집행미국 시장
전문
# Tick Data Dataset


# Tick Data Dataset

Market-by-order tick data for order book analysis and microstructure research.

| Property | Value |
|----------|-------|
| **Providers** | Databento (MBO), NASDAQ FTP (ITCH) |
| **Asset Class** | US Equities |
| **Frequency** | Tick (microsecond) |
| **Coverage** | Point-in-time |
| **Size** | MBO: ~500 MB, ITCH: ~5 GB/day |
| **API Key** | MBO: `DATABENTO_API_KEY` (**PAID**), ITCH: None |
| **Loaders** | `load_mbo_data()`, `load_nasdaq_itch()` |

**NOTE**: MBO data is expensive (~$0.50/symbol/day). ITCH is free but requires parsing.

```python
"""Tick Data - download, explore, and update workflow."""

import os
from datetime import datetime, timedelta
from pathlib import Path

import polars as pl
from dotenv import load_dotenv

# Load environment variables
load_dotenv()
```

## 1. Configuration

Two tick data sources are available:

| Source | Type | Cost | Parsing |
|--------|------|------|---------|
| Databento MBO | Order book events | ~$0.50/symbol/day | Pre-processed |
| NASDAQ ITCH | Raw exchange feed | Free | Requires parsing |

**MBO (Market-By-Order)**: Clean order book events - add, modify, cancel, trade.
**ITCH**: Native NASDAQ TotalView-ITCH 5.0 messages - requires binary parsing.

```python
print("=== Tick Data Configuration ===")
print()
print("Databento MBO:")
print("  Provider: Databento API (XNAS.ITCH dataset)")
print("  Cost: ~$0.45-0.50 per symbol per day")
print("  Default symbols: NVDA, SPY, TSLA")
print("  Format: Pre-processed Parquet")
print()
print("NASDAQ ITCH:")
print("  Provider: NASDAQ FTP (free)")
print("  Cost: Free (requires parsing)")
print("  File size: 4-6 GB per day (compressed)")
print("  Format: Binary (requires parsing)")
```

## 2. API Key Setup

### Databento MBO (Paid)

Databento requires a paid API key. New accounts receive $125 free credit.

1. Sign up at [Databento](https://databento.com/signup)
2. Navigate to **API Keys** in your dashboard
3. Add to your `.env` file:

```bash
DATABENTO_API_KEY=db-your-api-key-here
```

### NASDAQ ITCH (Free)

No API key required. Data is freely available from NASDAQ's FTP server.

```python
# Verify Databento API key
databento_key = os.getenv("DATABENTO_API_KEY")
if databento_key:
    print(f"DATABENTO_API_KEY: {databento_key[:8]}... (configured)")
else:
    print("DATABENTO_API_KEY: Not configured")
    print("  Get key at: https://databento.com/signup ($125 free credit)")

print()
print("NASDAQ ITCH: No API key required (free FTP access)")
```

## 3. Download Data

**WARNING**: MBO data is expensive. Always estimate cost before downloading!

```python
# Default configuration
DEFAULT_MBO_SYMBOLS = ["NVDA", "SPY", "TSLA"]
DEFAULT_MBO_DAYS = 10


def get_trading_dates(start_date: str, end_date: str) -> list[str]:
    """Generate list of trading dates (weekdays) between start and end."""
    start = datetime.strptime(start_date, "%Y-%m-%d")
    end = datetime.strptime(end_date, "%Y-%m-%d")

    dates = []
    current = start
    while current <= end:
        if current.weekday() < 5:  # Skip weekends
            dates.append(current.strftime("%Y%m%d"))
        current += timedelta(days=1)

    return dates


def estimate_mbo_cost(
    symbols: list[str] | None = None,
    dates: list[str] | None = None,
    start_date: str | None = None,
    end_date: str | None = None,
) -> dict:
    """Estimate MBO download cost.

    Args:
        symbols: List of symbols (default: NVDA, SPY, TSLA)
        dates: List of dates in YYYYMMDD format
        start_date: Start date (YYYY-MM-DD) if not providing dates list
        end_date: End date (YYYY-MM-DD) if not providing dates list

    Returns:
        Cost estimate dictionary
    """
    if symbols is None:
        symbols = DEFAULT_MBO_SYMBOLS

    if dates is None and start_date and end_date:
        dates = get_trading_dates(start_date, end_date)
    elif dates is None:
        # Default: 10 trading days
        end = datetime(2024, 11, 15)
        start = end - timedelta(days=20)
        dates = get_trading_dates(start.strftime("%Y-%m-%d"), end.strftime("%Y-%m-%d"))
        dates = dates[-DEFAULT_MBO_DAYS:]

    cost_per_symbol_day = 0.50
    total_symbol_days = len(symbols) * len(dates)
    estimated_cost = total_symbol_days * cost_per_symbol_day

    print("=== MBO Cost Estimate ===")
    print(f"Symbols: {symbols}")
    print(f"Days: {len(dates)} ({dates[0]} to {dates[-1]})")
    print(f"Total symbol-days: {total_symbol_days}")
    print(f"Cost per symbol-day: ${cost_per_symbol_day:.2f}")
    print("")
    print(f"ESTIMATED COST: ${estimated_cost:.2f}")

    return {
        "symbols": symbols,
        "num_days": len(dates),
        "total_symbol_days": total_symbol_days,
        "estimated_cost_usd": estimated_cost,
    }


def download_mbo_data(
    symbols: list[str] | None = None,
    dates: list[str] | None = None,
    start_date: str | None = None,
    end_date: str | None = None,
    dry_run: bool = True,  # Default True for safety!
    force: bool = False,
):
    """Download MBO tick data from Databento.

    Args:
        symbols: List of symbols (default: NVDA, SPY, TSLA)
        dates: List of dates in YYYYMMDD format
        start_date: Start date (YYYY-MM-DD) if not providing dates list
        end_date: End date (YYYY-MM-DD) if not providing dates list
        dry_run: If True, show cost estimate without downloading (DEFAULT: True)
        force: If True, re-download even if data exists
    """
    import databento as db

    from utils import ML4T_DATA_PATH

    if symbols is None:
        symbols = DEFAULT_MBO_SYMBOLS

    if dates is None and start_date and end_date:
        dates = get_trading_dates(start_date, end_date)
    elif dates is None:
        end = datetime(2024, 11, 15)
        start = end - timedelta(days=20)
        dates = get_trading_dates(start.strftime("%Y-%m-%d"), end.strftime("%Y-%m-%d"))
        dates = dates[-DEFAULT_MBO_DAYS:]

    output_dir = ML4T_DATA_PATH / "equities" / "market" / "microstructure" / "market_by_order"

    print("=== MBO Data Download ===")
    print(f"Symbols: {symbols}")
    print(f"Dates: {len(dates)} days ({dates[0]} to {dates[-1]})")
    print(f"Output: {output_dir}")

    # Always show cost estimate
    estimate = estimate_mbo_cost(symbols, dates)

    if dry_run:
        print("\n[DRY RUN] Would download:")
        for symbol in symbols:
            print(f"  {symbol}: {len(dates)} days")
        print("\nSet dry_run=False to actually download.")
        print(f"WARNING: This will cost ~${estimate['estimated_cost_usd']:.2f}")
        return

    # Verify API key
    api_key = os.getenv("DATABENTO_API_KEY")
    if not api_key:
        raise ValueError("DATABENTO_API_KEY not set. See API Key Setup section.")

    client = db.Historical(api_key)

    print(f"\nDownloading {len(symbols)} symbols x {len(dates)} days...")
    for symbol in symbols:
        symbol_dir = output_dir / symbol
        symbol_dir.mkdir(parents=True, exist_ok=True)

        for date in dates:
            output_file = symbol_dir / f"{date}.parquet"

            if output_file.exists() and not force:
                print(f"  Skipping {symbol}/{date} (exists)")
                continue

            print(f"  {symbol}/{date}...", end=" ", flush=True)
            try:
                date_str = f"{date[:4]}-{date[4:6]}-{date[6:8]}"
                data = client.timeseries.get_range(
                    dataset="XNAS.ITCH",
                    schema="mbo",
                    symbols=[symbol],
                    start=f"{date_str}T00:00:00",
                    end=f"{date_str}T23:59:59",
                )

                df = data.to_df()
                if len(df) > 0:
                    pl_df = pl.from_pandas(df.reset_index())
                    pl_df.write_parquet(output_file)
                    print(f"OK ({len(df):,} events)")
                else:
                    print("No data")
            except Exception as e:
                print(f"ERROR: {e}")

    print("\n=== Complete ===")
    print(f"Data saved to: {output_dir}")


def download_nasdaq_itch(date: str = "01302020", dry_run: bool = False):
    """Download NASDAQ ITCH sample data.

    Args:
        date: Date in MMDDYYYY format (default: 01302020)
        dry_run: If True, show what would be downloaded
    """
    from ml4t.data.providers.nasdaq_itch import ITCHSampleProvider

    from utils import ML4T_DATA_PATH

    output_dir = ML4T_DATA_PATH / "equities" / "market" / "microstructure" / "nasdaq_itch" / "raw"

    print("=== NASDAQ ITCH Download ===")
    print("Source: NASDAQ TotalView-ITCH 5.0")
    print("URL: https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/")
    print(f"Date: {date}")
    print(f"Output: {output_dir}")
    print()
    print("WARNING: Files are 4-6 GB each!")
    print("   Download may take 30-60 minutes.")
    print("   Requires parsing before use (see Chapter 4 notebooks)")

    if dry_run:
        print("\n[DRY RUN] Would download:")
        print(f"  {date}.NASDAQ_ITCH50.gz (~5 GB)")
        print("\nSet dry_run=False to actually download.")
        return

    output_dir.mkdir(parents=True, exist_ok=True)
    provider = ITCHSampleProvider(download_path=output_dir)

    print(f"\nDownloading {date}.NASDAQ_ITCH50.gz...")
    output_path = provider.download(date_or_filename=date, output_path=output_dir)

    print("\n=== Complete ===")
    print(f"Data saved to: {output_path}")
    print("\nNext steps:")
    print("  1. Parse the binary data using the Rust parser or Python")
    print("  2. See Chapter 4 notebooks for parsing examples")
```

### Estimate MBO Cost (ALWAYS DO THIS FIRST!)

```python
# Estimate cost for default symbols and dates
estimate_mbo_cost()
```

### Download MBO Data

```python
# Dry run (default) - shows what would be downloaded
download_mbo_data(dry_run=True)
```

```python
# Uncomment to actually download (after reviewing cost!)
# download_mbo_data(dry_run=False)
```

### Download NASDAQ ITCH

```python
# Dry run
download_nasdaq_itch(dry_run=True)
```

```python
# Uncomment to download (4-6 GB, takes 30-60 minutes)
# download_nasdaq_itch(dry_run=False)
```

## 4. Load and Explore

Once downloaded, use the loaders throughout the book:

```python
from data import load_mbo_data, load_nasdaq_itch
```

### MBO Data (Databento)

```python
# Load MBO data (if available)
mbo = load_mbo_data(symbols=["NVDA"], start_date="2024-11-04", end_date="2024-11-04")

print(f"Shape: {mbo.shape}")
print(f"Columns: {mbo.columns}")
print(f"Memory: {mbo.estimated_size('mb'):.1f} MB")
```

```python
# Preview
mbo.head(10)
```

```python
# Event type distribution
if "action" in mbo.columns:
    print("Event types:")
    mbo.group_by("action").len().sort("len", descending=True)
```

### NASDAQ ITCH

```python
# Load ITCH data (if available and parsed)
itch = load_nasdaq_itch()

print(f"Shape: {itch.shape}")
print(f"Memory: {itch.estimated_size('mb'):.1f} MB")
```

```python
# Message type distribution
if "msg_type" in itch.columns:
    print("Message types:")
    itch.group_by("msg_type").len().sort("len", descending=True)
```

## 5. Data Profile

```python
from ml4t.data.storage.data_profile import get_profile_path, load_profile

from utils import ML4T_DATA_PATH

microstructure = ML4T_DATA_PATH / "equities" / "market" / "microstructure"

for name, data_path in [
    ("MBO", microstructure / "market_by_order"),
    ("ITCH", microstructure / "nasdaq_itch"),
]:
    profile_path = get_profile_path(data_path)
    profile = load_profile(profile_path)
    if profile is None:
        print(f"No {name} profile at {profile_path}")
    else:
        print(f"=== {name} Profile ===")
        print(f"Written by {profile.source}")
        print(profile.summary())

print(
    "\nThe two downloaders in this directory write raw vendor captures and do not go\n"
    "through ml4t.data.storage.data_profile, so neither dataset carries a profile today.\n"
    "Nothing in this notebook writes one either."
)
```

## 6. Loader Options

Both loaders support filtering by symbols and dates.

### MBO Data

```python
# Load specific symbol and date
nvda = load_mbo_data(symbols=["NVDA"], start_date="2024-11-04", end_date="2024-11-04")
print(f"NVDA Nov 4: {nvda.shape}")
```

```python
# Load multiple symbols
multi = load_mbo_data(symbols=["NVDA", "SPY"], start_date="2024-11-04", end_date="2024-11-04")
print(f"NVDA + SPY: {multi.shape}")
```

### NASDAQ ITCH

```python
# Load all parsed messages
itch_day = load_nasdaq_itch()
print(f"ITCH messages: {itch_day.shape}")
```

```python
# Load specific message types
# trades = load_nasdaq_itch(message_types=["P", "Q"])
```

## 7. Documentation

### Databento MBO

- [Databento Documentation](https://databento.com/docs/)
- [XNAS.ITCH Dataset](https://databento.com/docs/datasets/xnas-itch)
- [MBO Schema](https://databento.com/docs/schemas/mbo)

### MBO Event Types

| Action | Description |
|--------|-------------|
| `A` | Add order to book |
| `M` | Modify existing order |
| `C` | Cancel order |
| `T` | Trade execution |
| `F` | Order fully filled |

### NASDAQ ITCH

- [NASDAQ TotalView-ITCH](https://www.nasdaq.com/docs/TotalView-ITCH-5-0.pdf)
- [NASDAQ FTP](https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/)

### ITCH Message Types

| Type | Description |
|------|-------------|
| `S` | System event |
| `R` | Stock directory |
| `A` | Add order (no MPID) |
| `F` | Add order (with MPID) |
| `E` | Order executed |
| `C` | Order executed with price |
| `X` | Order cancel |
| `D` | Order delete |
| `U` | Order replace |
| `P` | Trade (non-cross) |
| `Q` | Cross trade |

### Data Quality Notes

- **MBO**: Pre-processed, clean timestamps, consistent schema
- **ITCH**: Raw binary, requires parsing, timestamp normalization needed
- **MBO cost**: ~$0.45-0.50 per symbol per day
- **ITCH size**: 4-6 GB per day (compressed)

## 8. Updating Data

Tick data is typically downloaded **point-in-time** for specific analysis periods.

### MBO Updates

```python
# Download specific dates (ALWAYS estimate first!)
estimate_mbo_cost(symbols=["AAPL"], start_date="2024-12-01", end_date="2024-12-05")
download_mbo_data(symbols=["AAPL"], start_date="2024-12-01", end_date="2024-12-05", dry_run=False)
```

### ITCH Updates

```python
# Download specific date
download_nasdaq_itch(date="01152025", dry_run=False)
```

### Cost Considerations

| Data Type | Cost | Recommendation |
|-----------|------|----------------|
| MBO | ~$0.50/symbol/day | Download only needed periods |
| ITCH | Free | Download as needed (5 GB/day) |

**Best practice**: Use ITCH for broad market analysis, MBO for specific symbols.

## Summary

| Source | Cost | Format | Use Case |
|--------|------|--------|----------|
| Databento MBO | ~$0.50/symbol/day | Pre-processed Parquet | Specific symbol analysis |
| NASDAQ ITCH | Free | Raw binary (5 GB/day) | Broad market analysis |

**Loaders**:
- `load_mbo_data(symbols, start_date, end_date)` - Databento MBO
- `load_nasdaq_itch(date)` - NASDAQ ITCH

**Primary use**: Order book reconstruction, microstructure research, market making analysis.
**Critical**: Always run `estimate_mbo_cost()` before downloading MBO data!

출처의 라이선스에 따라 출처를 표시하고 전문을 공개합니다. 라이선스: MIT

이 요약은 원문을 바탕으로 Stratmill의 리서치 에이전트가 작성했으며, 원문을 복사한 것이 아닙니다.