Stochastische Diskontfaktoren für ETF: Renditebewertung mit Querschnittsdaten

# ETFs: a model that prices the cross-section instead of predicting it


# ETFs: a model that prices the cross-section instead of predicting it

The three members before this one share a shape. [`11a_pca`](11a_pca.ipynb),
[`11b_ipca`](11b_ipca.ipynb) and
[`11c_conditional_autoencoder`](11c_conditional_autoencoder.ipynb) all split the problem in two:
estimate what each fund is exposed to, estimate what those exposures earned, multiply. They
differ in how the first half is written down - not at all, a linear map, a network - and in
nothing else.

**The stochastic discount factor does not split it.** It asks for a single series that prices
every fund at once: a weighting of the cross-section whose covariance with each fund's return
accounts for that fund's average return. That is a statement about *no arbitrage* rather than
about forecasting, and it is fitted by driving pricing errors toward zero rather than by
minimising a forecast error. The two networks it trains - one producing the discount-factor
weights from the cross-section, one producing each fund's exposure to it - are trained against
that pricing objective.

A predicted return still comes out, which is why this notebook sits beside the others and is
scored the same way. But the number being optimised is not the number being reported, and that
gap is the thing to hold on to while reading the result.

**It is also the only member that reads macro data.** The other four see the fund panel and the
feature matrix. This one additionally takes eleven macro series as context for the
discount factor, which raises a question none of the others have to answer: what was actually
knowable on the decision date. Section 2 is where that contract is read back out of the resolved
specification rather than asserted here.

**The checkpoint schedule is phased, and all of it is published.** Training runs an
unconditional stage and then a conditional one, and `case_studies/config/sdf/sdf.yaml` declares
saves inside each. An earlier version of this notebook fixed the last of them in advance,
showed that a different checkpoint scored better on validation, and reported the fixed one
anyway. That is the right instinct about the trap - picking the checkpoint with the best
validation IC is choosing after seeing the answer - but the wrong remedy, because it discards
models the case study paid to train and leaves the choice to be made again, undocumented,
downstream. Publishing the whole schedule lets [`14_backtest`](14_backtest.ipynb) select on
validation backtest Sharpe with every candidate visible and the rule stated. **Selection happens
there, not here.**

**Learning objectives.** By the end of this notebook you will be able to:

- Say what a pricing objective asks for that a forecasting objective does not.
- Read a phased training schedule and work out how many prediction sets a label will owe.
- Read a point-in-time data contract out of a resolved specification and say what it rules out.
- Say why the macro context belongs inside the model's identity rather than beside it.

**Book reference**: Chapter 14, Section 14.7 (The stochastic discount factor and the supervised
autoencoder models). Chapter 6, Section 6.7 (Search accounting and run logging) introduces the
run log this notebook writes to.

**Prerequisites**: [`03_financial_features`](03_financial_features.ipynb) and
[`04_model_based_features`](04_model_based_features.ipynb) for the feature matrix,
[`05_evaluation`](05_evaluation.ipynb) for the walk-forward folds, and
[`11c_conditional_autoencoder`](11c_conditional_autoencoder.ipynb), which is the two-stage model
this one abandons.

**What it writes**: one training run per label and one complete validation prediction set per
label and checkpoint, in `run_log/registry.db` and under `run_log/training/` and
`run_log/predictions/`, grouped under a population named for this model. The family splits across
five notebooks, so each publishes its own population rather than one shared one.

```python
"""Fit the declared ETF stochastic-discount-factor population on the walk-forward folds."""

# Load PyTorch's bundled CUDA runtime before other ML4T libraries.
import torch  # noqa: F401

# isort: split
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots

from case_studies.research import (
    declared_labels,
    load_model_configs,
    model_requests,
    open_study,
    primary_label,
    resolved_model_plan,
    run_model_population,
)
from utils.style import COLORS, show_plotly_with_alt
```

```python
LABELS: list[str] = []
EXECUTION_TIER = "canonical"
WORKSPACE: str = ""
PREVIEW_REDUCTIONS: dict = {}
POPULATION_NAME = ""
SUPERSEDES_POPULATION: str = ""
DEVICE: str = ""

MODEL_NAME = "sdf"
```

```python
study = open_study("etfs", execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
```

## 1. Which labels, and what the configuration says

Every label whose training menu declares `latent_factors:` is fitted, and both do: `fwd_ret_21d`,
the total return over the 21 trading days after the decision date, and `fwd_ret_5d`, the same
thing over five. The pricing objective is written against the label's returns, so these are two
separate fits rather than one fit scored twice. `LABELS` restricts the run to a subset when you
want one.

```python
declared_labels(study, "latent_factors")
```

`case_studies/config/sdf/sdf.yaml` declares the estimator and its schedule, and the schedule has
more structure here than in the two autoencoders. `n_epochs_unc` trains the unconditional stage,
`n_epochs_cond` the conditional one that follows it, and `checkpoint_epochs` lists the saves
inside a stage. The resolved plan in section 2 turns those into the actual list of checkpoints,
which is what decides how many prediction sets each label owes - the count is not the length of
the declared list, because a save inside the unconditional stage and the same offset inside the
conditional one are two different states of the model.

```python
configs = load_model_configs(
    study,
    "latent_factors",
    labels=LABELS or None,
    config_names=[MODEL_NAME],
)
configs.select("label", "config_name", "params")
```

`LABELS` narrows what is fitted, and a narrowed run declares a different set of members than the
canonical population does. A population is immutable once written, so such a run must publish
under its own name: on a fresh workspace it would otherwise register an incomplete snapshot under
the canonical one, and where the full population already exists the registry refuses it.

The comparison is against this model's own declared rows rather than the whole `latent_factors`
catalog. The family is split across five notebooks and each publishes one model, so the complete
catalog is five times what this one publishes, and comparing against it would report every
canonical run as narrowed.

The device is checked in the same cell. Two networks trained on a GPU and the same two trained on
a CPU accumulate their sums in different orders and reach different weights, so the device is
part of what the fitted model is and is recorded inside the computation's identity rather than
beside it. `PUBLISHED_DEVICE` is the device this population was fitted on; the runner refuses to
substitute a CPU for a requested GPU rather than publishing a different model under the published
name, so on a machine with no NVIDIA card this notebook stops at the next cell. Set
`DEVICE="cpu"` and pass a `POPULATION_NAME` to fit the same schedule there.

```python
PUBLISHED_DEVICE = "cuda"
device = DEVICE or PUBLISHED_DEVICE
print(f"training device: {device}")

declared = load_model_configs(study, "latent_factors", config_names=[MODEL_NAME])
narrowed = set(configs.get_column("label")) != set(declared.get_column("label"))
if (narrowed or device != PUBLISHED_DEVICE) and not POPULATION_NAME:
    raise ValueError(
        f"this run fits {configs.height} of the {declared.height} declared labels on device "
        f"{device!r}, which is not the complete declared catalog on {PUBLISHED_DEVICE!r}, so it "
        "cannot publish the canonical population; pass POPULATION_NAME to give it its own"
    )
```

## 2. Binding the declarations to the data

A menu entry says which estimator to fit. It does not say which fund-date pairs have both a
feature row and a label, or where the walk-forward folds fall. **Resolving** a request goes and
finds that: it reads the label, feature and macro files, computes the fold boundaries from the
walk-forward parameters in `config/setup.yaml`, and works out the exact set of rows each fit is
expected to predict. It fits nothing, so the plan can be read before any training starts.

Four things to check in it:

- **`feature_count`, `eligible_entities` and `eligible_rows` agree across the rows.** They are
  the width of the networks' input, the number of funds in the cross-section, and the fund-date
  pairs to be predicted. A row that differs is a label measured on a different sample.
- **`folds` is the same on both rows**, and equals the number of walk-forward splits
  [`05_evaluation`](05_evaluation.ipynb) established.
- **`validation_start` and `validation_end` bracket the development sample.** The held-out tail
  is scored once, at the end of the case study; any of it visible here would mean it had been
  used to choose something.
- **`checkpoints`** is the resolved schedule length, and it is the number to multiply by the row
  count to get the candidate models this notebook is about to create.

Each row also carries a `training_hash`: the identity of that computation, derived from
everything that can change its result. [`RUN_LOG.md`](../RUN_LOG.md#identity) sets out what goes
into one and what follows from it.

```python
requests = model_requests(
    study,
    configs,
    execution_tier=EXECUTION_TIER,
    overrides={"device": device},
    preview_reductions=PREVIEW_REDUCTIONS,
)
resolved = tuple(request.resolve() for request in requests)

plan = resolved_model_plan(resolved)
plan.select(
    "label",
    "config_name",
    "feature_count",
    "eligible_entities",
    "eligible_rows",
    "folds",
    "checkpoints",
    "validation_start",
    "validation_end",
)
```

The plan is also where a short population is visible. A run that fitted fewer labels than the
menu declares would still print an IC table and still register its rows; what it would not do is
announce the gap.

```python
planned_pairs = set(zip(plan["label"], plan["config_name"], strict=True))
requested_pairs = set(zip(configs["label"], configs["config_name"], strict=True))
if planned_pairs != requested_pairs:
    raise RuntimeError(
        "the plan does not match the loaded SDF menu; "
        f"missing {sorted(requested_pairs - planned_pairs)}, "
        f"unexpected {sorted(planned_pairs - requested_pairs)}"
    )
print(f"{len(requested_pairs)} label-configuration pairs")
```

### The macro contract, read back from what was resolved

The discount factor takes macro context, and the point-in-time question that raises is not
rhetorical: a macro series as it reads today is not what it read on a decision date years ago,
because the agencies that publish these numbers revise them. Training on the revised series would
let the model condition on information nobody had.

The contract below is read out of the resolved specification rather than restated here, so it is
what actually entered the identity:

- **`source`** says the values are ALFRED *initial releases* - each series as it was first
  published, not as it now reads.
- **`availability_lag_days`** and **`alignment`** say a value dated *t* becomes usable on *t+1*
  and is then carried backward, so a same-day close cannot reach a same-day decision.
- **`coverage_start`** is where the panel begins, which is set by the last series to start rather
  than the first.
- **`resolved_fold_digest`** is the content of the macro values each fold actually saw. It is
  inside the training identity, so a macro panel that changed underneath would register as a
  different model rather than overwriting this one.

**What this does not buy.** The financial feature matrix still contains yield-curve variables
built from finalized FRED series, so the run as a whole is not a vintage-clean live estimate. The
macro contract covers the macro context and nothing else, and the feature matrix's exact content
is pinned by the modeling-input digest rather than by a release policy.

```python
macro = resolved[0].spec["computation"]["macro_context"]
if macro is None:
    raise RuntimeError("the SDF request resolved without a macro context")

pl.DataFrame(
    {
        "field": ["source", "policy", "version", "availability_lag_days", "alignment"]
        + ["coverage_start", "series", "resolved_fold_digest"],
        "value": [
            str(macro["source"]),
            str(macro["policy"]),
            str(macro["version"]),
            str(macro["availability_lag_days"]),
            str(macro["alignment"]),
            str(macro["coverage_start"]),
            str(len(macro["series"])),
            str(macro["resolved_fold_digest"]),
        ],
    }
)
```

## 3. Fitting the population

`run_model_population` runs every resolved request. For one request it walks the folds, and on
each one:

1. takes the fund-dates inside that fold's training window, together with the macro values that
   were available on those dates,
2. trains the unconditional stage, then the conditional one that adds the macro context, by
   gradient descent on the pricing errors, writing the weights to disk at each declared save,
3. applies each saved set of weights to the validation dates to get each fund's exposure to the
   discount factor there, and turns that into a predicted return.

Step 3 is what makes one training run produce several results. The fold predictions are
concatenated into one series per checkpoint covering the whole validation period, and each
becomes its own registered prediction set with its own identity.

**What the call publishes is a population**: a named, immutable list of the prediction sets it
will produce, computed from the resolved specifications and written down before the first fit.
Afterwards every member must exist and be complete, which is what makes the downstream comparison
well defined - [`14_backtest`](14_backtest.ipynb) backtests this population, not whatever
predictions happen to be in the registry - and it is why a configuration that raises fails the
whole call rather than publishing a population one member short.

`SUPERSEDES_POPULATION` names the population hash this run replaces. A population is the set of
prediction identities, so anything that moves a training identity - a changed macro panel as
much as a changed epoch schedule - produces a different set under the same name, and the
registry refuses to write it without being told which snapshot it supersedes. It is empty here
because `etfs-sdf-validation-v1` has one generation and this run is it: there is no earlier
snapshot to replace. It stays empty on a re-run too, because reproducing a member list already
published under that name is not a replacement. Fill it in when you have changed something that
moves an identity and want the new set to take the name; the error raised on the attempt tells
you which hash to name. A reduced-scale run passes it empty whatever the default is: a
population produced under a reduction is thrown away with the workspace it was written to, so it
has no lineage to extend.

```python
population_name = POPULATION_NAME or f"etfs-{MODEL_NAME}-validation-v1"
execution, population = run_model_population(
    study,
    resolved,
    population_name=population_name,
    supersedes=SUPERSEDES_POPULATION or None,
)

published_sets = sum(len(run.predictions) for run in execution.runs)
print(f"{len(execution.runs)} configurations, {published_sets} prediction sets")
print(f"population {population.name}: {len(population.members)} members")
```

A second run of this notebook trains nothing. Every identity is re-derived from the inputs, the
registry already holds the matching rows and the saved weights, and the runner returns the stored
result rather than training again - so re-running it unchanged costs the time it takes to read
the data. The latent-factor runner reports no per-fold fitted-or-reused breakdown, unlike the
linear and boosted families, so the counts above are of what the population holds rather than of
what this particular run computed.

### Running configurations of your own

The published run log is read-only. To add runs, open the study against a workspace, which holds
its own registry and artifacts and reads the same labels, features and macro panel:

```python
study = open_study("etfs", workspace="~/ml4t-experiments")
configs = load_model_configs(study, "latent_factors", labels=["fwd_ret_5d"], config_names=["sdf"])
requests = model_requests(study, configs, overrides={"device": "cuda"})
resolved = tuple(request.resolve() for request in requests)
execution, population = run_model_population(study, resolved, population_name="my-sdf-v1")
```

To change the architecture or the schedule, edit `case_studies/config/sdf/sdf.yaml`. That changes
this configuration's identity, so its result registers as a new row beside the old one rather
than replacing it - and that includes the phase lengths and `checkpoint_epochs`, which decide how
many members the population has. Give the run its own `population_name`: a name refers to one set
of members permanently, and reusing it for a different set raises.
[`RUN_LOG.md`](../RUN_LOG.md#running-your-own-configurations) covers the rest.

## 4. What came out

One row per label and checkpoint, read back from the registry. `ic_mean` is the **information
coefficient**: on each validation date, rank the funds by the model's prediction, rank them by
the return they went on to earn, correlate the two rankings, and average that daily correlation
over the validation period.

`ic_n_days` is how many validation dates produced a defined correlation, and it decides which
rows are comparable with each other. A model that has settled into assigning nearly the same
exposure to every fund on a date gives that date no spread to rank, and the date drops out of the
average - so that checkpoint's `ic_mean` is an average over a different sample from its
neighbours'. `full_coverage` marks the rows measured on every date their own label offers, and
the curves below are restricted to those. **Coverage is judged within a label**, because a
21-day forward return runs out of window earlier than a five-day one and one global maximum would
mark the whole 21-day grid incomplete for a reason that has nothing to do with the models.

The published catalog is checked against the population planned before fitting rather than
against its own row count, because a run that lost a member would otherwise report a shorter
table and nothing else.

```python
catalog = execution.catalog_rows.select(
    "label",
    "config_name",
    "complete",
    "checkpoint_kind",
    "checkpoint_value",
    "ic_mean",
    "ic_std",
    "ic_n_days",
    "n_folds",
    "training_hash",
    "prediction_hash",
).sort("label", "checkpoint_value")

if set(catalog.get_column("prediction_hash")) != set(population.members):
    raise RuntimeError("the published catalog differs from the population planned before fitting")
if catalog.filter(~pl.col("complete")).height:
    raise RuntimeError("a partial SDF prediction set cannot pass to backtesting")

catalog = catalog.with_columns(
    full_coverage=pl.col("ic_n_days") == pl.col("ic_n_days").max().over("label")
)

primary = primary_label(study)
present = sorted(set(catalog.get_column("label")))
# The primary label leads when it was fitted. A subset run that leaves it out orders the panels by
# whichever label it did fit rather than by one that is not there.
panel_labels = [label for label in [primary] if label in present] + [
    label for label in present if label != primary
]
print(f"{catalog.height} candidate models across {len(panel_labels)} labels")
print(f"at {catalog.n_unique('checkpoint_value')} checkpoints each")
catalog.select("label", "checkpoint_value", "ic_mean", "ic_std", "ic_n_days", "full_coverage")
```

### What more training does

Each line traces one label's out-of-sample IC across the phased schedule. The horizontal axis is
cumulative epochs, so the first point is the end of the unconditional stage and the rest are
saves inside the conditional one - the step from the first to the second is where the macro
context enters, and it is the only step on the axis that changes what the model is allowed to
see rather than only how long it has trained.

A line that rises and then falls has an interior optimum: the model was still learning, then
began fitting the training window at the expense of the validation folds. A line that wanders
around zero without trend never had anything to learn, and its highest point is wherever the
noise happened to peak. Both produce a respectable-looking maximum, which is why the maximum is
not what a checkpoint is judged on and why the whole schedule is published rather than the best
of it.

One panel per label, each with its own vertical scale, because a horizon with something to learn
and one without would be averaged into a single indistinct band if they shared axes.

```python
curves = catalog.filter("full_coverage").sort("label", "checkpoint_value")
fig = make_subplots(
    rows=len(panel_labels),
    cols=1,
    shared_xaxes=True,
    vertical_spacing=0.08,
    subplot_titles=[
        f"{label} ({'primary' if label == primary else 'variant'})" for label in panel_labels
    ],
)
for row, label in enumerate(panel_labels, start=1):
    series = curves.filter(pl.col("label") == label)
    fig.add_trace(
        go.Scatter(
            x=series.get_column("checkpoint_value").to_list(),
            y=series.get_column("ic_mean").to_list(),
            mode="lines+markers",
            name=label,
            showlegend=False,
            line=dict(color=COLORS["blue"], width=2),
            marker=dict(size=6, color=COLORS["blue"]),
        ),
        row=row,
        col=1,
    )
    fig.add_hline(y=0, line_width=1, line_dash="dash", line_color=COLORS["neutral"], row=row, col=1)
    fig.update_yaxes(title_text="Mean IC (validation)", row=row, col=1)
fig.update_xaxes(title_text="Cumulative training epoch", row=len(panel_labels), col=1)
fig.update_layout(
    title="Out-of-sample ranking accuracy across the phased schedule",
    height=320 * len(panel_labels),
    width=800,
    margin=dict(t=90),
)
# Read off the frame rather than asserted, so the alt text stays true of the next run.
_spans = "; ".join(
    f"{label} from {curves.filter(pl.col('label') == label).get_column('ic_mean').min():.3f} "
    f"to {curves.filter(pl.col('label') == label).get_column('ic_mean').max():.3f}"
    for label in panel_labels
)
show_plotly_with_alt(
    fig,
    "Line chart of mean validation information coefficient against cumulative training epoch, one "
    "panel per label, each with its own vertical scale and a dashed zero line. Counted from the "
    f"frame: {_spans}.",
)
```

### What one checkpoint's number averages

Each row of the catalog is one number over the whole validation period, and the eight folds
behind it are trained on different windows and scored on different years. The table below
recomputes each fold's mean IC from the published predictions of one checkpoint, so the
disagreement between folds measuring the same model can be compared against the differences
between checkpoints in the curve above.

The checkpoint shown is the last of the schedule. That is where training stopped, not a
selection: reading these eight numbers at the checkpoint with the highest `ic_mean` and reporting
them as the model's would be the choice this notebook publishes the whole schedule to avoid.

```python
published = {result.hash: result for run in execution.runs for result in run.predictions}
last_checkpoint = catalog.get_column("checkpoint_value").max()


def _fold_ic(prediction_hash: str) -> pl.DataFrame:
    """Mean daily rank correlation of prediction against realised return, within each fold."""
    return (
        published[prediction_hash]
        .load()
        .group_by("fold", "timestamp")
        .agg(pl.corr("prediction", "actual", method="spearman").alias("ic"))
        .group_by("fold")
        .agg(
            pl.col("ic").mean().alias("fold_ic"),
            pl.len().alias("validation_dates"),
        )
        .sort("fold")
    )


by_fold = {
    row["label"]: _fold_ic(row["prediction_hash"])
    for row in catalog.filter(pl.col("checkpoint_value") == last_checkpoint).iter_rows(named=True)
}
for label in panel_labels:
    ic = by_fold[label].get_column("fold_ic")
    # See 11b: polars returns None for the mean of an empty column and for the standard
    # deviation of a single row, and a narrowed population produces both.
    if ic.len() == 0:
        print(f"{label} at cumulative epoch {last_checkpoint}: no fold scored, so there is no IC")
        continue
    spread = (
        f"standard deviation across folds {ic.std():.4f}"
        if ic.len() > 1
        else "one fold only, so no spread across folds"
    )
    print(
        f"{label} at cumulative epoch {last_checkpoint}: mean {ic.mean():+.4f}, "
        f"{spread}, "
        f"{(ic < 0).sum()} of {ic.len()} folds negative"
    )

by_fold[panel_labels[0]]
```

## 5. What to notice

**The macro step is the whole story on the primary label, and it runs the other way on the
five-day one.** Checkpoint 256 is where the unconditional stage ends, and on `fwd_ret_21d` it
scores **-0.0070** - the model orders the cross-section slightly worse than a coin. The four
conditional saves that follow run **+0.0132, +0.0223, +0.0379, +0.0370**. Every bit of ranking
accuracy this notebook publishes on its primary label appears after the macro context enters, at
the one step on the axis that changes what the model is allowed to see rather than only how long
it has trained. On `fwd_ret_5d` the same step is a loss: 256 scores **+0.0394**, the highest of
its five, and the four conditional checkpoints sit **below** it at +0.0091, +0.0152, +0.0233 and
+0.0238. Conditioning on eleven macro series helps the 21-day horizon and hurts the 5-day one,
which is not a result that survives being averaged into one number for the model.

**Pricing the cross-section did not beat splitting the problem in two.** Against the members
fitted on the same panel over the same folds, this one's best primary-label checkpoint
(**+0.0379**) sits below [`11b_ipca`](11b_ipca.ipynb)'s single **+0.0413** and well below
[`11c_conditional_autoencoder`](11c_conditional_autoencoder.ipynb)'s **+0.0722** at the end of its
schedule. On the five-day label its best (+0.0394, from the unconditional stage) is near IPCA's
+0.0349 and under the autoencoder's +0.0457. It clears [`11a_pca`](11a_pca.ipynb)'s -0.0326 on
the primary label, so conditioning still bought something over conditioning on nothing - just
less than either two-stage member bought, at four times the training. That is the trade this
member exists to price, and here the answer is negative.

**The checkpoint curve is not evidence for an epoch, and the arithmetic says so plainly.** The
five primary-label checkpoints span 0.0449 of IC end to end. The eight fold ICs behind the last of
them - +0.005, +0.027, +0.165, +0.159, +0.077, +0.021, -0.140, -0.017 - have a standard deviation
across folds of **0.0989**, more than twice that span. The apparent best checkpoint, 1024, beats
1280 by 0.0009, a fiftieth of the disagreement between folds measuring the same model. Nothing
here distinguishes an epoch worth stopping at from an epoch that happened to score well on eight
folds, which is why all five are published and why nothing downstream may select among them on
validation IC. The dispersion is also no better than the neighbours': 0.0989 here against 0.092
for the conditional autoencoder and 0.099 for IPCA.

**No checkpoint collapsed, and that had to be checked rather than assumed.** `ic_n_days` is 1,995
on every primary-label row and 2,011 on every five-day row - the full count each label offers - so
`full_coverage` holds for all ten published models. A model that had settled into pricing every
fund alike on a date would have dropped that date out of its average and left one row measured on
a different sample from its neighbours'.

**Known limitations.** The architecture, the factor count and the two-phase schedule are declared
in `case_studies/config/sdf/sdf.yaml` rather than searched, so nothing here says this is the right
SDF - only what this one did. The macro contract is point-in-time by construction, and its
coverage starts **2010-11-23** while the panel starts in 2006, so the earliest training windows
carry less macro history than the later ones and the conditional stage is not learning from the
same amount of context in every fold. Two of eight folds are negative on the primary label and
three on the five-day. The panel narrows from 96 funds in the first fold to 90 in the last. The
published result is a CUDA result, and a reader who refits on CPU gets a different identity
registered beside it rather than these numbers. And every number here is measured on validation
folds that have been read many times over by the time a case study reaches this notebook.

**Next**: [`11e_supervised_autoencoder`](11e_supervised_autoencoder.ipynb) is the family's own
control - it keeps [`11c`](11c_conditional_autoencoder.ipynb)'s network and bottleneck while
dropping the factor interpretation, so what it does not achieve is what the structure was worth.
[`13_model_analysis`](13_model_analysis.ipynb) is where all five are read against the other
families.