Zeitreihenprognosebibliotheken für SPY-Kurse vergleichen

# The same forecast, written six ways


# The same forecast, written six ways

**Docker image**: `ml4t-gpu`

Every notebook in this chapter has been written directly against PyTorch: a model
class, a training loop, an evaluation block. That is the right way to learn an
architecture and a poor way to try five of them. This notebook writes the same
univariate forecast through several libraries and compares the *experience* -
how many lines, what has to be installed, what the interface constrains.

**sktime** is the one to understand, because it is not a model library. It is an
interface - `fit`, `predict`, `update` - that delegates to NeuralForecast,
PyTorch Forecasting, HuggingFace and classical methods, so swapping an LSTM for
Chronos or for ARIMA is a constructor change rather than a rewrite. **Darts**
takes the other approach: its own `TimeSeries` container and its own model
implementations.

**What the table measures is effort, not accuracy.** The rows do not share a target
scale or a test set, so the error column cannot be read across them - which is
stated where the table appears and marked in the table's own columns. A last-value
baseline is included precisely so that the one comparison that *is* like-for-like
is on the page.

**Learning objectives**:
- Write the same forecast against a raw framework and against a wrapper, and count
  what the wrapper actually saved.
- Read a comparison table and identify which columns can be compared across rows
  and which cannot.
- Say what a rank correlation means when the target is a price level rather than a
  return.
- Decide when the interface constraint is worth the boilerplate it removes.

**Book Reference**: Chapter 13, Section 13.7 (A practical framework)

**Prerequisites**: ETF price data via the canonical `load_etfs()` loader

```python
"""Library Landscape - compare raw PyTorch vs sktime-wrapped forecasting implementations."""

import logging
import tempfile
import time
import warnings
from datetime import datetime

# pytorch_lightning emits this at import time, which `from darts...` below triggers, so
# the filter has to be installed before the import rather than alongside the others.
warnings.filterwarnings(
    "ignore",
    message=r".*LeafSpec.*is deprecated",
    module=r".*pytorch_lightning.*",
)

import numpy as np
import pandas as pd
import plotly.graph_objects as go
import polars as pl
import torch
import torch.nn as nn
from darts import TimeSeries
from darts.models import RNNModel
from ml4t.diagnostic.metrics import pooled_ic
from sktime.forecasting.chronos import ChronosForecaster
from sktime.forecasting.patch_tst import PatchTSTForecaster

from data import load_etfs
from utils.reproducibility import set_global_seeds
from utils.style import (  # COLORS activates the ml4t Plotly template on import
    COLORS,
    show_plotly_with_alt,
)
```

Three of the libraries below write progress banners to standard error - the accelerator
they picked, a suggestion to install a logging integration, a deprecation inside
`transformers` reached through sktime's Chronos wrapper. None of it reports a problem
and none of it is this notebook's output, so each is quieted by name at its own logger
rather than by a blanket filter. Anything these libraries raise as a warning still
reaches the render.

```python
logging.getLogger("pytorch_lightning").setLevel(logging.ERROR)
logging.getLogger("lightning.pytorch").setLevel(logging.ERROR)
logging.getLogger("darts").setLevel(logging.ERROR)
logging.getLogger("transformers").setLevel(logging.ERROR)
```

```python
SEED = 42
LOOKBACK = 60
HORIZON = 5
HIDDEN_SIZE = 64
EPOCHS = 30
BATCH_SIZE = 32
START_DATE = "2015-01-01"
```

```python
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Device: {DEVICE}")

set_global_seeds(SEED)
```

## Data: SPY Close Prices

We use a single univariate series (SPY daily closes) so every library sees
identical input. We prepare two representations: NumPy sequences for raw
PyTorch, and a pandas Series with business-day index for sktime/Darts.

```python
etf_df = load_etfs()
start_dt = datetime.fromisoformat(START_DATE)

spy = (
    etf_df.filter((pl.col("symbol") == "SPY") & (pl.col("timestamp") >= start_dt))
    .sort("timestamp")
    .select(["timestamp", "close"])
)

prices = spy["close"].to_numpy().astype(np.float32)
n_seq = len(prices) - LOOKBACK - HORIZON + 1
split_idx = int(n_seq * 0.8)
train_price_end = split_idx + LOOKBACK + HORIZON
price_mean = prices[:train_price_end].mean()
price_std = prices[:train_price_end].std()
prices_norm = (prices - price_mean) / price_std

print(f"SPY: {len(prices):,} daily observations")
```

The order of the two steps above matters. The train/test boundary is fixed in
sequence space *first*, and the normalisation statistics are then computed from
prices up to `train_price_end` only - the last price index any training sequence can
reach, its input window plus the forecast horizon. Normalising with the full
series' mean and standard deviation would fold test-period price levels into the
training inputs, which is a leak that leaves no trace in any split.

### PyTorch Sequences (Normalized)

```python
def create_sequences(data, lookback, horizon):
    """Create (input, target) pairs for univariate forecasting."""
    X, y = [], []
    for i in range(len(data) - lookback - horizon + 1):
        X.append(data[i : i + lookback])
        y.append(data[i + lookback : i + lookback + horizon].mean())
    return np.array(X, dtype=np.float32), np.array(y, dtype=np.float32)


X, y = create_sequences(prices_norm, LOOKBACK, HORIZON)
# split_idx was fixed above (in price space) so the normalization stays train-only
X_train, X_test = X[:split_idx], X[split_idx:]
y_train, y_test = y[:split_idx], y[split_idx:]

X_train_t = torch.FloatTensor(X_train).unsqueeze(-1).to(DEVICE)
y_train_t = torch.FloatTensor(y_train).unsqueeze(-1).to(DEVICE)
X_test_t = torch.FloatTensor(X_test).unsqueeze(-1).to(DEVICE)

print(f"Sequences: train={len(X_train):,}, test={len(X_test):,}")
```

### sktime / Darts Series (Prices with DatetimeIndex)

```python
spy_pd = spy.to_pandas()
spy_series = spy_pd.set_index("timestamp")["close"]
spy_series = spy_series.sort_index()
spy_series.index = pd.DatetimeIndex(spy_series.index)

# Reindex to business-day frequency, forward-filling gaps (holidays etc.)
# Libraries like sktime/NeuralForecast and Darts require a regular frequency.
bday_idx = pd.bdate_range(start=spy_series.index.min(), end=spy_series.index.max(), freq="B")
spy_series = spy_series.reindex(bday_idx).ffill().dropna()
spy_series.index.freq = "B"

sk_split = int(len(spy_series) * 0.8)
y_train_sk = spy_series.iloc[:sk_split]
y_test_sk = spy_series.iloc[sk_split : sk_split + HORIZON]
fh_sk = list(range(1, HORIZON + 1))
```

### Evaluation Helper

**The IC here is not the IC in the rest of the chapter.** Notebooks 04 through 10
compute a *cross-sectional* rank correlation: on each date, do the predicted ranks
of the funds match the realised ranks? There is one series here, so there is no
cross-section, and what this helper computes is the rank correlation between
predicted and actual values across time.

On a price level that number is close to one for anything that tracks the level,
including a forecast that just repeats the last observation - the baseline below
demonstrates it. Read the `ic` column as a check that a model is not producing
nonsense, and read MSE for anything else.

```python
def evaluate(y_true, y_pred):
    """Compute MSE and Spearman IC."""
    mse = float(np.mean((y_true - y_pred) ** 2))
    ic = pooled_ic(y_pred, y_true) if len(y_true) > 5 else float("nan")
    return {"mse": round(mse, 6), "ic": round(ic, 4) if np.isfinite(ic) else None}


results = []  # Collector for final comparison
```

### The forecast to beat

Before any library: forecast the next `HORIZON`-day mean as the last observed
price. It costs one line, it is what every model below has to improve on, and on a
price level it is hard to beat - which is exactly why it belongs in the table.

The IC column needs the same warning. On a trending price level, any forecast that
tracks the level correlates with the outcome almost perfectly, so a rank correlation
near one is a statement about the series, not about the model. The baseline row is
what makes that visible.

```python
preds_naive = X_test[:, -1]
m_naive = evaluate(y_test, preds_naive)
results.append(
    {
        "Model": "Last value",
        "Library": "None",
        "Lines": "~1",
        "Fit + Predict (s)": 0.0,
        **m_naive,
        "Trainable": False,
        "Target Scale": "normalized",
        "Evaluation Points": len(y_test),
    }
)
print(f"Last-value baseline: MSE={m_naive['mse']}, IC={m_naive['ic']}")
```

## Part 1: Raw PyTorch LSTM

The "full control" baseline: model definition, training loop, and evaluation
require roughly 50 lines of boilerplate.

```python
class LSTMForecaster(nn.Module):
    """Minimal LSTM for univariate forecasting."""

    def __init__(self, hidden_size, num_layers=2):
        super().__init__()
        self.lstm = nn.LSTM(1, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, 1)

    def forward(self, x):
        out, _ = self.lstm(x)
        return self.fc(out[:, -1, :])
```

```python
model = LSTMForecaster(HIDDEN_SIZE).to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

start = time.time()
for epoch in range(EPOCHS):
    model.train()
    indices = torch.randperm(len(X_train_t))
    epoch_loss, n_batches = 0.0, 0
    for i in range(0, len(X_train_t), BATCH_SIZE):
        batch_idx = indices[i : i + BATCH_SIZE]
        loss = criterion(model(X_train_t[batch_idx]), y_train_t[batch_idx])
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        n_batches += 1
    if (epoch + 1) % 10 == 0:
        print(f"  Epoch {epoch + 1}/{EPOCHS}: loss={epoch_loss / n_batches:.6f}")
model.eval()
with torch.no_grad():
    preds_pt = model(X_test_t).cpu().numpy().flatten()
pytorch_time = time.time() - start

m = evaluate(y_test, preds_pt)
results.append(
    {
        "Model": "LSTM",
        "Library": "Raw PyTorch",
        "Lines": "~50",
        "Fit + Predict (s)": round(pytorch_time, 1),
        **m,
        "Trainable": True,
        "Target Scale": "normalized",
        "Evaluation Points": len(y_test),
    }
)
print(f"Raw PyTorch LSTM: {pytorch_time:.1f}s, MSE={m['mse']}, IC={m['ic']}")
print(
    f"  against the last-value baseline: MSE={m_naive['mse']}, IC={m_naive['ic']} "
    f"on the same {len(y_test)} points"
)
```

The raw PyTorch LSTM requires defining the model class, training loop, and
evaluation - approximately 50 lines. Full control over architecture and training
dynamics, at the cost of that boilerplate.

Compare the two lines printed above before reading anything else in this notebook as
a result. Both are on the same scale and the same test points, which is the only
such comparison here, and the baseline needed no training at all.

## Part 2: sktime + NeuralForecast LSTM

sktime wraps NeuralForecast's LSTM: three lines replace the 50-line training
loop. The underlying recurrent architecture is equivalent.

**Dependency note**: sktime's neural forecasters require `neuralforecast`, which
depends on `ray`, and Python 3.14 wheels for ray are still pending
([ray-project/ray#56434](https://github.com/ray-project/ray/issues/56434)). This
section describes the call rather than running it. Note also that recent sktime
releases renamed the wrapper's `hidden_size` argument to `encoder_hidden_size`.

The shape of the call, when it can be made, is
`NeuralForecastLSTM(freq=..., input_size=..., max_steps=..., encoder_hidden_size=...)`
followed by `fit` and `predict` - the whole of Part 1's model class and training
loop replaced by a constructor. That is the argument for the wrapper, and this
environment is also the argument against depending on one: the code above cannot
run here at all.

## Part 3: sktime + PyTorch Forecasting N-BEATS

N-BEATS (Section 13.2) via sktime's PyTorch Forecasting wrapper. Same
`fit()`/`predict()` API, different architecture underneath.

**Dependency note**: sktime's PyTorch Forecasting wrapper requires
`pytorch-forecasting`, and the last attempt also hit a wrapper API mismatch around
`max_prediction_length` and `max_encoder_length`. This section describes the call
rather than running it.

The call would be `PytorchForecastingNBeats(max_prediction_length=...,
max_encoder_length=..., max_epochs=..., trainer_kwargs=...)`, then the same `fit`
and `predict` as every other forecaster here - which is the point of the interface:
swapping an LSTM for the N-BEATS of `02_nbeats_interpretable` changes an import and
a constructor and nothing about the data pipeline.

It is also the second wrapper in a row that this notebook cannot execute, which is
information about wrappers rather than an aside.

## Part 4: sktime PatchTST

PatchTST (Section 13.5) via sktime's HuggingFace integration. Supports
full training, fine-tuning, and zero-shot modes.

**Dependency note**: PatchTST uses sktime's HuggingFace backend, not
`neuralforecast`, so it runs in the current Python 3.14 environment. The
first run downloads model artifacts and trains into a temporary output
directory.

```python
forecaster = PatchTSTForecaster(
    fit_strategy="full",
    config={
        "context_length": LOOKBACK,
        "prediction_length": HORIZON,
        "patch_length": 12,
        "num_hidden_layers": 2,
        "d_model": HIDDEN_SIZE,
    },
    training_args={
        "output_dir": tempfile.mkdtemp(prefix="patchtst_"),
        "num_train_epochs": min(EPOCHS, 10),
        "per_device_train_batch_size": BATCH_SIZE,
        "logging_strategy": "no",
    },
)
start = time.time()
forecaster.fit(y_train_sk, fh=fh_sk)
y_pred = forecaster.predict()
t = time.time() - start

m = evaluate(y_test_sk.values, y_pred.values[:HORIZON])
results.append(
    {
        "Model": "PatchTST",
        "Library": "sktime/HuggingFace",
        "Lines": "~8",
        "Fit + Predict (s)": round(t, 1),
        **m,
        "Trainable": True,
        "Target Scale": "raw_price",
        "Evaluation Points": HORIZON,
    }
)
print(f"sktime PatchTST: {t:.1f}s, MSE={m['mse']}, IC={m['ic']}")
```

PatchTST via HuggingFace requires slightly more configuration (patch size,
training arguments) but still follows the `fit`/`predict` pattern.

## Part 5: sktime Chronos (Zero-Shot Foundation Model)

Chronos (Section 13.6) requires **no training**. The `fit()` call registers
history; all computation happens in `predict()`. This is the ultimate
rapid-prototyping workflow: instant baseline with zero boilerplate.

**Dependency note**: Chronos also uses sktime's HuggingFace backend and runs
in the current Python 3.14 environment. The first run downloads the Chronos
checkpoint before generating zero-shot forecasts.

```python
forecaster = ChronosForecaster("amazon/chronos-t5-tiny")
start = time.time()
forecaster.fit(y_train_sk)
y_pred = forecaster.predict(fh=fh_sk)
t = time.time() - start

m = evaluate(y_test_sk.values, y_pred.values)
results.append(
    {
        "Model": "Chronos (tiny)",
        "Library": "sktime/HuggingFace",
        "Lines": "~3",
        "Fit + Predict (s)": round(t, 1),
        **m,
        "Trainable": False,
        "Target Scale": "raw_price",
        "Evaluation Points": HORIZON,
    }
)
print(f"sktime Chronos: {t:.1f}s (zero-shot), MSE={m['mse']}, IC={m['ic']}")
```

Zero-shot inference eliminates training entirely. Accuracy depends on domain
match with the pretraining corpus (see Section 13.6 for foundation model details).

## Part 6: Darts LSTM (Comparison Point)

Darts is a popular alternative with its own `TimeSeries` container and API.
This provides one comparison point outside the sktime ecosystem.

Two details of the Darts container are worth stating rather than leaving in the
code. It needs the reindexed business-day series so its declared frequency is
consistent, and its `TimeSeries` defaults to float64 - which the Lightning trainer's
`accelerator="auto"` cannot use when it selects MPS on Apple Silicon, since MPS
rejects float64 tensors. Casting to float32 runs identically on CPU, CUDA and MPS.

```python
spy_bday_df = spy_series.reset_index()
spy_bday_df.columns = ["timestamp", "close"]
ts = TimeSeries.from_dataframe(
    spy_bday_df, time_col="timestamp", value_cols="close", freq="B"
).astype(np.float32)
ts_train = ts[:sk_split]
ts_test = ts[sk_split : sk_split + HORIZON]

# Re-asserted here rather than only at import: something between the import cell and this
# one puts the LeafSpec deprecation back, and a filter has to be in force where the
# warning is raised, not merely where it was first installed.
warnings.filterwarnings(
    "ignore",
    message=r".*LeafSpec.*is deprecated",
    module=r".*pytorch_lightning.*",
)

# No output_chunk_length: Darts' RNNModel forecasts one step and rolls it forward, so it
# overrides any value passed here. `predict(HORIZON)` still returns HORIZON steps.
darts_model = RNNModel(
    model="LSTM",
    input_chunk_length=LOOKBACK,
    training_length=LOOKBACK + HORIZON,
    hidden_dim=HIDDEN_SIZE,
    n_epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    random_state=SEED,
    pl_trainer_kwargs={
        "enable_progress_bar": False,
        "enable_model_summary": False,
        "logger": False,
        "accelerator": "auto",
    },
)
start = time.time()
darts_model.fit(ts_train)
y_pred_darts = darts_model.predict(HORIZON)
t = time.time() - start

m = evaluate(ts_test.values().flatten()[:HORIZON], y_pred_darts.values().flatten()[:HORIZON])
results.append(
    {
        "Model": "LSTM",
        "Library": "Darts",
        "Lines": "~10",
        "Fit + Predict (s)": round(t, 1),
        **m,
        "Trainable": True,
        "Target Scale": "raw_price",
        "Evaluation Points": HORIZON,
    }
)
print(f"Darts LSTM: {t:.1f}s, MSE={m['mse']}, IC={m['ic']}")
```

Darts provides a self-contained ecosystem with its own `TimeSeries` container.
The `random_state` parameter ensures reproducibility - a feature not all
wrapper APIs expose.

## Comparison Table

Every row starts from the same data, SPY close prices, and the table collects
whichever approaches ran. Of the six library demos attempted, the ones that
complete in this environment are raw PyTorch, sktime's PatchTST and Chronos, and
Darts; sktime's NeuralForecast wrapper is blocked by `ray` on Python 3.14 and its
PyTorch Forecasting wrapper by an upstream API mismatch. The last-value baseline
needs no library at all.

**Read `Lines` and `Fit + Predict (s)` across rows. Do not read `mse` across rows.**
The `Target Scale` and `Evaluation Points` columns say why: raw PyTorch and the
baseline fit a *normalized* target and are scored over hundreds of rolling windows,
while sktime and Darts work on raw price levels and are scored on a single
`HORIZON`-step forecast. The scale gap alone moves the number by orders of
magnitude, and five evaluation points would not support a comparison even if the
scales matched.

The one pair that *is* comparable is the baseline against raw PyTorch: same scale,
same test points. That comparison is the only accuracy statement this notebook
supports, and it is worth reading before the rest of the table.

```python
if results:
    comparison = pl.DataFrame(results)
else:
    comparison = None
    print("No results collected -- check library installations.")

comparison
```

### Boilerplate: lines of code per approach

Implementation effort is what this table measures cleanly. The raw-PyTorch path
needs a model class and a training loop; each wrapper is a constructor plus `fit`
and `predict`. Line counts are approximate, but they are counted the same way for
every row, which is more than the error column can say.

```python
if results:
    _labels = [f"{r['Model']}<br>({r['Library']})" for r in results]
    _lines = [int(str(r["Lines"]).lstrip("~")) for r in results]
    _colors = [
        COLORS["slate"]
        if r["Library"] == "None"
        else COLORS["amber"]
        if "PyTorch" in r["Library"]
        else COLORS["blue"]
        for r in results
    ]
    fig = go.Figure(
        go.Bar(x=_labels, y=_lines, marker_color=_colors, text=_lines, textposition="outside")
    )
    fig.update_layout(
        title="Approximate lines of code per approach",
        yaxis_title="Approximate lines of code",
        showlegend=False,
    )
    show_plotly_with_alt(
        fig,
        "A bar chart of approximate lines of code, one bar per approach, labelled with "
        "the model and the library. The raw PyTorch bar is amber, the library-wrapped "
        "bars navy, and the no-library baseline slate.",
    )
```

## When to Use Each Approach

The section text (Section 13.7) compares library *capabilities* (panel support,
foundation models). This table focuses on *implementation experience*:

| Approach | Implementation Effort | Flexibility | Best For |
|----------|----------------------|-------------|----------|
| **Raw PyTorch** | ~50 lines, full boilerplate | Full control over everything | Custom architectures, cross-sectional ranking, research |
| **sktime** | ~3-8 lines | Constrained to `fit`/`predict` API | Rapid prototyping, backend swapping, standardized benchmarks |
| **Darts** | ~10 lines | Moderate (own `TimeSeries` container) | Probabilistic forecasting, self-contained ecosystem |
| **Chronos / TSFMs** | ~3 lines, zero training | None (zero-shot only) | Instant baselines, cold-start scenarios |

sktime is not a model library. It is a single interface - `fit`, `predict`,
`update` - delegating to NeuralForecast (LSTMs, TCN), PyTorch Forecasting (N-BEATS,
TFT), HuggingFace (PatchTST, Chronos) and classical methods (ETS, ARIMA), so an
LSTM, a foundation model and an ARIMA can be benchmarked without touching the data
pipeline. What this notebook also shows is the cost of that indirection: two of the
backends could not be installed or called here, and the interface does not help
with a backend that is not there.

```python
if results:
    _libraries = [r for r in results if r["Library"] != "None"]
    print(f"Library approaches that ran: {len(_libraries)} of 6 attempted")
    trainable = [r for r in _libraries if r["Trainable"]]
    zero_shot = [r for r in _libraries if not r["Trainable"]]
    if trainable:
        fastest = min(trainable, key=lambda r: r["Fit + Predict (s)"])
        print(
            f"Fastest to fit and predict: {fastest['Model']} ({fastest['Library']}) "
            f"at {fastest['Fit + Predict (s)']}s"
        )
    if zero_shot:
        print(f"Zero-shot models: {len(zero_shot)} (no training required)")
```

## Key takeaways

1. **Two of the table's columns can be compared across rows, and the error column
   cannot.** Lines of code is one; the other is wall-clock time, which is measured
   over the same span for every row - the fit and the prediction together - because
   that is the only boundary all of them share. Chronos does no fitting at all, so
   separating the two would leave its cell empty and the column unreadable. Error is
   the one that cannot be compared: the raw-PyTorch row is a normalized target scored
   over a long rolling test set, the sktime and Darts rows are raw price levels
   scored on a single `HORIZON`-step forecast. The scale gap moves the number by
   orders of magnitude and the sample gap moves its variance. `Target Scale` and
   `Evaluation Points` are in the table so the mismatch cannot be missed.
2. **The last-value baseline is the row to read first.** It shares its scale and its
   test points with the raw-PyTorch row, so those two are genuinely comparable - and
   it needed no training. On a price level, that is a demanding thing to beat.
3. **A rank correlation on a price level is not a measure of forecasting skill.**
   Any forecast that tracks the level scores near one, including the baseline. That
   is why this notebook's `ic` column is a diagnostic and the multi-asset notebooks
   compute a *cross-sectional* IC instead, which asks a question the level cannot
   answer for you.
4. **sktime is an interface, not a model library.** One `fit`/`predict` surface
   delegating to NeuralForecast, PyTorch Forecasting, HuggingFace and classical
   methods is what lets an LSTM, Chronos and ARIMA be benchmarked without touching
   the data pipeline. Darts makes the opposite trade: its own container and its own
   models, self-contained rather than delegating.
5. **What a wrapper costs is the thing it does not expose.** A custom loss, a
   non-standard training schedule or a cross-sectional target is where the
   `fit`/`predict` surface stops helping, which is the case the rest of this chapter
   is written in raw PyTorch for.
6. **Installability is part of the comparison.** Not every wrapper ran: the
   printed count says how many of the attempted demos completed in this environment,
   and the sections above say what blocked each one.

**Next**: `12_case_study_insights` leaves single-series demonstrations behind and
aggregates what these architectures did across the book's case studies under
walk-forward validation.
![notebook output](figures/p1_1.png)