Воспроизводимые каталоги экспериментов на основе реестра исследований

# Experiment Tracking with MLflow and the Case-Study Registry


# Experiment Tracking with MLflow and the Case-Study Registry

**Chapter 26: MLOps and Governance**
**Docker image**: `ml4t`
**Book Reference**: Chapter 26, Sections 26.1, 26.2, 26.4, 26.6
**Prerequisites**: Basic model evaluation, validation workflows, and Chapter 25 deployment verification.

**Learning Objectives**:
- Read a run log as an experiment tracker: what each row records, how a run is identified,
  and how a result is traced back to the files that produced it.
- Rebuild a searchable catalog of every model run from the tables the pipeline writes, and
  see from it what was tried and what has not been.
- Log the same catalog into MLflow and check that its queries return the same answers, so
  that what a tracker buys can be separated from what the discipline buys.

An **experiment tracker** answers three questions after the fact: what was run, what came
out, and can it be reproduced. The case studies in this book already answer them, through a
SQLite database of runs beside a directory of content-addressed artifact bundles. This
notebook reads that registry first, because seeing the answers assembled by hand is what
makes clear which part MLflow automates.

**Content-addressed** means a run's identifier is a hash of the specification that produced
it, so the same specification always yields the same identifier. That is what makes a run
log idempotent: re-running a pipeline either lands on the row that already exists or
produces a different hash, and there is no third case where two rows describe the same run.

```python
"""Experiment Tracking with MLflow and the Case-Study Registry: one catalog, read two ways."""
```

## Settings

`PRIMARY_LABEL` and `SECONDARY_LABEL` name the two return horizons whose runs the catalog
reads. A case study declares several, and reading two of them is enough to show that the
catalog groups by label rather than pooling across horizons.

`MAX_MLFLOW_RUNS` caps how many catalog rows are logged into MLflow. Logging every run would
be the production behaviour and would make this notebook slow; the cap is why the sample is
constructed deliberately below rather than taken off the top.

`PARITY_TOLERANCE` is how far apart the two systems' metrics may be. Both read the same
stored value, so anything above float noise means one of them is reporting a different run.

`BACKTEST_PANEL_ROWS` and `HASH_PREFIX_CHARS` govern how much of the backtest table the
figure shows and how much of each hash labels a bar.

```python
CASE_STUDY_ID = "us_equities_panel"
PRIMARY_LABEL = "fwd_ret_1d"
SECONDARY_LABEL = "fwd_ret_5d"
MAX_MLFLOW_RUNS = 50
PARITY_TOLERANCE = 1e-8
BACKTEST_PANEL_ROWS = 6
HASH_PREFIX_CHARS = 4
```

```python
import json
import sqlite3
import warnings
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import polars as pl
from IPython.display import display

from utils.paths import get_case_study_dir, get_output_dir
from utils.style import COLORS, FIGSIZE, add_message_title, ml4t_palette, show_with_alt, zero_line

# Named, not blanket: a bare ignore would also hide the convergence and numerical
# warnings a reader needs to see.
warnings.filterwarnings("ignore", category=FutureWarning, module="mlflow")

CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
REGISTRY_PATH = CASE_DIR / "run_log" / "registry.db"
TRAINING_DIR = CASE_DIR / "run_log" / "training"
PREDICTIONS_DIR = CASE_DIR / "run_log" / "predictions"

print("Experiment Tracking with MLflow and the Case-Study Registry")
print("=" * 60)
```

## 1. What the registry records

A run log has to carry five things before it can answer the three questions above:
provenance, the data and evaluation protocol, the configuration, the artifacts, and the
decision gates that were applied. The case-study pipeline in Chapters 11 to 16 records them
across three levels, each one a stage of the same experiment:

$$\textrm{training run} \rightarrow \textrm{prediction set} \rightarrow \textrm{backtest run}$$

A training run is a fitted model under one configuration. A prediction set is what that
model produced on one split. A backtest run is what a strategy built from those predictions
did. Each level is identified by a hash of its own canonical specification, and each carries
the identifier of the level above it, so a Sharpe ratio at the bottom resolves to the exact
configuration and data at the top.

```python
from case_studies.utils.registry import REGISTRY_SCHEMA_SQL, canonical_json, compute_hash

print("Registry schema (tables and indexes):")
for line in REGISTRY_SCHEMA_SQL.strip().split("\n"):
    stripped = line.strip()
    if stripped.startswith("CREATE TABLE") or stripped.startswith("CREATE INDEX"):
        print(f"  {stripped.split('(')[0]}")
```

### The hash is the identifier

Canonicalizing the specification before hashing is what makes the identifier stable: the
same settings serialize to the same bytes whatever order they were written in, so two runs
of the same experiment collide by design rather than accumulating as separate rows.

```python
example_config = {
    "family": "gbm",
    "label": "fwd_ret_1d",
    "config_name": "leaves_31_mae",
    "seed": 42,
}
example_hash = compute_hash(canonical_json(example_config))
print(f"Config:  {canonical_json(example_config)}")
print(f"Hash:    {example_hash}")
print(f"Stable:  {example_hash == compute_hash(canonical_json(example_config))}")
```

MLflow's tracking database holds the same categories under different names: parameters,
metrics, artifact locations and run lineage. The difference is the interface, and the
sections below load the registry directly first so that the comparison in section 7 is
between two answers rather than between an answer and an assumption.

## 2. Load the registry tables

The registry stores the essential experiment-tracking payload across five
tables: `training_runs`, `prediction_sets`, `prediction_metrics`,
`backtest_runs`, and `backtest_metrics`.

```python
def query_table(query: str, params: tuple[object, ...] = ()) -> pd.DataFrame:
    with sqlite3.connect(REGISTRY_PATH) as conn:
        return pd.read_sql_query(query, conn, params=params)
```

```python
training_runs = query_table(
    """
    SELECT training_hash, family, label, config_name, created_at, spec_json
    FROM training_runs
    WHERE label IN (?, ?)
    """,
    (PRIMARY_LABEL, SECONDARY_LABEL),
)

backtest_runs = query_table(
    """
    SELECT br.backtest_hash, br.prediction_hash, br.spec_json, br.created_at,
           tr.label, tr.family, tr.config_name
    FROM backtest_runs br
    JOIN prediction_sets ps ON br.prediction_hash = ps.prediction_hash
    JOIN training_runs tr ON ps.training_hash = tr.training_hash
    WHERE tr.label IN (?, ?)
    """,
    (PRIMARY_LABEL, SECONDARY_LABEL),
)

print(f"Training runs loaded: {len(training_runs):,}")
print(f"Backtest runs loaded: {len(backtest_runs):,}")
```

## 3. Build the searchable experiment catalog

Join training runs with their prediction metrics to reconstruct the
experiment catalog that would normally appear in an experiment tracker UI.

```python
run_catalog = query_table(
    """
    SELECT
        tr.training_hash,
        tr.family,
        tr.label,
        tr.config_name,
        tr.created_at,
        ps.prediction_hash,
        ps.split,
        pm.ic_mean_daily AS ic_mean
    FROM training_runs tr
    JOIN prediction_sets ps ON tr.training_hash = ps.training_hash
    LEFT JOIN prediction_metrics pm
        ON ps.prediction_hash = pm.prediction_hash
    WHERE tr.label IN (?, ?)
      AND ps.split = 'validation'
      AND pm.ic_mean_daily IS NOT NULL
      AND pm.ic_n_days > 0
    ORDER BY tr.label, tr.family, pm.ic_mean_daily DESC
    """,
    (PRIMARY_LABEL, SECONDARY_LABEL),
)

print(f"Validation prediction sets: {len(run_catalog):,}")
run_catalog.head(10)
```

```python
best_validation = (
    run_catalog.sort_values("ic_mean", ascending=False)
    .groupby(["label", "family"], as_index=False)
    .first()
    .sort_values(["label", "ic_mean"], ascending=[True, False])
)
best_validation
```

The catalog answers the first question an experiment review asks: which configurations were
tried, and how did each one score.

What it does not answer is which one to deploy, and the distinction is worth holding onto.
The information coefficient is the rank correlation between a model's scores and the
returns they predicted, so ordering by it says which model ranked the cross-section best -
a description of the predictions. This pipeline selects on validation backtest Sharpe
instead, because a model can rank well and still produce a portfolio nobody would hold once
turnover and costs are charged. The catalog ranks; the selection happens elsewhere.

## 4. What the predictions did as a portfolio

The backtest tables carry what happened when a strategy was built from each prediction set,
which is the evidence the section above cannot supply. Reading it here is also how a
catalog reports a gap: a prediction set with no backtest row is a run that was fitted and
never evaluated as a strategy, and a tracker that only listed what exists would not show
that.

The query is restricted to the equal-weight baseline, which the registry stores under
`stage='signal'`. Every other stage varies an allocator, a cost assumption or a risk overlay
on top of that baseline, and a comparison across stages is a comparison of overlays rather
than of models.

```python
backtest_pivot = query_table(
    """
    SELECT
        tr.family,
        tr.label,
        tr.config_name,
        br.backtest_hash,
        bm.sharpe,
        bm.cagr,
        bm.max_drawdown,
        bm.total_return
    FROM backtest_runs br
    JOIN prediction_sets ps ON br.prediction_hash = ps.prediction_hash
    JOIN training_runs tr ON ps.training_hash = tr.training_hash
    JOIN backtest_metrics bm ON br.backtest_hash = bm.backtest_hash
    WHERE tr.label IN (?, ?)
      AND br.stage = 'signal'
    """,
    (PRIMARY_LABEL, SECONDARY_LABEL),
)

if "sharpe" in backtest_pivot.columns:
    backtest_pivot = backtest_pivot.sort_values("sharpe", ascending=False)

backtest_pivot.head(10)
```

## 5. Trace one run back to its files

This is the third question - can it be reproduced - and it is the one a metric in a table
cannot answer on its own. The manifest resolves one catalog row to the specification that
configured the fit and the predictions it produced, both addressed by the hashes the row
carries. A missing file here is what a reproducibility claim looks like when it fails.

```python
selected = best_validation.iloc[0]
selected_training_hash = selected["training_hash"]
selected_prediction_hash = selected["prediction_hash"]

training_dir = TRAINING_DIR / selected_training_hash
prediction_dir = PREDICTIONS_DIR / selected_prediction_hash
run_log_display = Path("case_studies") / CASE_STUDY_ID / "run_log"

manifest = pd.DataFrame(
    [
        {
            "artifact": "training spec",
            "path": str(run_log_display / "training" / selected_training_hash / "spec.json"),
            "exists": (training_dir / "spec.json").exists(),
        },
        {
            "artifact": "predictions",
            "path": str(
                run_log_display / "predictions" / selected_prediction_hash / "predictions.parquet"
            ),
            "exists": (prediction_dir / "predictions.parquet").exists(),
        },
    ]
)
manifest
```

```python
if (training_dir / "spec.json").exists():
    spec = json.loads((training_dir / "spec.json").read_text())
    print(f"Selected training run: {selected_training_hash}")
    print(f"Family: {spec.get('family')}, Config: {spec.get('config_name')}")
    print(f"Label: {spec.get('label')}, Seed: {spec.get('seed')}")
    print(f"Identity version: {spec.get('identity_version')}, Tier: {spec.get('execution_tier')}")
    print(f"\nFull spec:\n{json.dumps(spec, indent=2)}")

if (prediction_dir / "predictions.parquet").exists():
    sample = pl.read_parquet(prediction_dir / "predictions.parquet").head(5)
    print(f"\nSample predictions ({selected_prediction_hash}):")
    display(sample)
```

## 6. The catalog as a picture

Two things in the catalog are read from their shape rather than off a row. The first is how a
family's scores are distributed: whether its best run sits inside a tight cluster or stands
well above a spread, and how far apart the families are relative to their own spreads. The
second is how far baseline Sharpe ranges across configurations.

What either one supports is a question, not an answer. A configuration's score is not an
independent draw - configurations within a family share features, folds and much of their
fitted state, so a cluster can be several views of one overfit and an isolated high score can
be a real improvement. Reading the distribution tells you which of those to go and check; the
check itself is a held-out evaluation, and the holdout is spent once.

```python
family_counts = run_catalog.groupby(["label", "family"]).size().reset_index(name="run_count")
```

```python
FAMILY_DISPLAY = {
    "deep_learning": "deep learning",
    "latent_factors": "latent factors",
    "linear": "linear",
    "tabular_dl": "tabular DL",
    "gbm": "GBM",
}


def family_label(name: str) -> str:
    return FAMILY_DISPLAY.get(name, name)
```

```python
fig, axes = plt.subplots(1, 2, figsize=FIGSIZE["dual_h"], constrained_layout=True)

# An empirical cumulative distribution rather than a strip plot: the families differ by an
# order of magnitude in run count, and a cumulative curve compares their shapes without the
# denser family covering the sparser one.
ax1 = axes[0]
family_order = run_catalog.groupby("family")["ic_mean"].median().sort_values().index.tolist()
family_colors = dict(zip(family_order, ml4t_palette(len(family_order), categorical=True)))
for family_name in family_order:
    values = run_catalog.loc[run_catalog["family"] == family_name, "ic_mean"].sort_values()
    share = np.arange(1, len(values) + 1) / len(values)
    ax1.step(
        values,
        share,
        where="post",
        linewidth=2,
        color=family_colors[family_name],
        label=f"{family_label(family_name)} (n={len(values)})",
    )
zero_line(ax1, axis="x")
ax1.set_ylim(0, 1.02)
add_message_title(
    ax1,
    "Distribution of validation information coefficient by family",
    subtitle="Share of a family's runs at or below each value",
)
ax1.set_xlabel("Mean daily IC")
ax1.set_ylabel("Share of runs")
ax1.legend(loc="lower right", fontsize=8, frameon=False)

ax2 = axes[1]
if backtest_pivot.empty or "sharpe" not in backtest_pivot.columns:
    ax2.text(
        0.5,
        0.5,
        "No baseline backtest rows in the registry\nfor the labels this notebook reads",
        ha="center",
        va="center",
        transform=ax2.transAxes,
    )
    ax2.set_xticks([])
    ax2.set_yticks([])
    add_message_title(ax2, "Baseline backtest Sharpe by configuration")
else:
    shown = backtest_pivot.head(BACKTEST_PANEL_ROWS).sort_values("sharpe")
    ax2.barh(
        shown["config_name"] + " / " + shown["backtest_hash"].str[:HASH_PREFIX_CHARS],
        shown["sharpe"].fillna(0),
        color=COLORS["positive"],
    )
    add_message_title(
        ax2,
        "Baseline backtest Sharpe by configuration",
        subtitle=f"Highest first, at most {BACKTEST_PANEL_ROWS} configurations",
    )
    ax2.set_xlabel("Annualized Sharpe ratio")
    ax2.tick_params(axis="y", labelsize=7)

show_with_alt(
    fig,
    "Two panels. Left: one step curve per model family giving the share of that family's "
    "runs whose mean daily validation information coefficient is at or below each value, "
    "against a dashed vertical line at zero, with the run count in each legend entry. "
    "Right: a horizontal bar chart of baseline backtest Sharpe by configuration, or a note "
    "that the registry holds no baseline backtest rows for these labels.",
)
```

### Best validation runs

```python
best_validation[["training_hash", "family", "label", "config_name", "ic_mean"]]
```

One row per family and label, holding the run with the highest mean daily validation IC in
that slice. Grouping before ranking is what keeps the table readable as the catalog grows:
an ungrouped ranking of every run tends to be one family's grid several times over.

### Training run counts by family and label

```python
family_counts
```

How many configurations each family was given, per label. This is the experiment budget, and
it is worth recording because a family tried a hundred ways and a family tried twice are not
comparable on their best score alone: the maximum of more draws is higher even when nothing
improved. How much higher depends on how correlated the configurations are, which the count
does not say - what the count does is make the asymmetry visible, where a catalog showing only
the top run per family hides it.

### Selected run manifest

```python
manifest
```

## 7. MLflow experiment tracking

Everything above was assembled with SQL written for this schema. MLflow supplies the same
operations as a library - `log_params`, `log_metrics`, `log_artifact` and `search_runs` -
against a schema it owns, with a web interface over it.

The catalog is logged into a local MLflow store below and then queried through its API. The
comparison at the end is the reason to do it: if both return the same runs in the same
order, then what the tracker supplies is the interface and the infrastructure, and the
properties that make either one trustworthy - deterministic hashing, a manifest that
resolves, a recorded specification - came from the pipeline that wrote the runs.

```python
import logging

import mlflow

# MLflow logs schema creation and experiment creation at INFO, on its own logger, and those
# lines land in the committed render as stderr. Raised by name to WARNING, so anything MLflow
# reports as a problem still reaches the reader.
logging.getLogger("mlflow").setLevel(logging.WARNING)

print(f"MLflow {mlflow.__version__} available")
```

### Set up a local file-backed tracking store

MLflow writes to a local directory with no server, which is what makes it usable inside a
notebook. The store goes under the chapter's output directory so nothing is written beside
the source.

```python
import shutil

MLFLOW_DIR = get_output_dir(26, "mlflow_tracking")
# A store left by an interrupted execution would return its runs from search_runs too, and
# the parity check below would compare against rows this run never logged.
shutil.rmtree(MLFLOW_DIR, ignore_errors=True)
MLFLOW_DIR.mkdir(parents=True, exist_ok=True)

mlflow.set_tracking_uri(f"sqlite:///{MLFLOW_DIR / 'mlflow.db'}")
experiment = mlflow.set_experiment(CASE_STUDY_ID)
print(f"Tracking URI : {MLFLOW_DIR}")
print(f"Experiment   : {experiment.name} (id={experiment.experiment_id})")
```

### Log catalog runs into MLflow

The sample is constructed rather than sliced. Every row that tops its family-label group
goes in first, and the remaining slots go to the highest-IC runs left over.

The construction is what makes the check at the end mean something. Under a plain top-N
slice, a group whose best run sits outside the global top `MAX_MLFLOW_RUNS` reaches the
manual catalog and not MLflow, and the comparison then reports a difference the sampling
created.

```python
def log_catalog_to_mlflow(catalog: pd.DataFrame, max_runs: int = MAX_MLFLOW_RUNS) -> int:
    """Log catalog rows as MLflow runs, returning how many were logged.

    A production pipeline logs one run as each training job finishes, so the cap exists
    only to keep this notebook fast.
    """
    logged = 0
    for row in catalog.itertuples(index=False):
        if logged >= max_runs:
            break
        ic = row.ic_mean
        if ic is None or (isinstance(ic, float) and pd.isna(ic)):
            continue

        with mlflow.start_run(run_name=row.training_hash):
            params = {"family": str(row.family), "label": str(row.label)}
            if row.config_name is not None:
                params["config_name"] = str(row.config_name)
            mlflow.log_params(params)

            metrics = {"ic_mean_daily": float(ic)}
            mlflow.log_metrics(metrics)

            spec_path = TRAINING_DIR / row.training_hash / "spec.json"
            if spec_path.exists():
                mlflow.log_artifact(str(spec_path))

        logged += 1
    return logged


leader_hashes = set(best_validation["training_hash"])
leaders = run_catalog[run_catalog["training_hash"].isin(leader_hashes)]
remaining = run_catalog[~run_catalog["training_hash"].isin(leader_hashes)].sort_values(
    "ic_mean", ascending=False
)
catalog_sample = pd.concat([leaders, remaining], ignore_index=True).head(MAX_MLFLOW_RUNS)
assert leader_hashes.issubset(set(catalog_sample["training_hash"]))
n_logged = log_catalog_to_mlflow(catalog_sample, max_runs=len(catalog_sample))
assert n_logged == len(catalog_sample)
print(f"Logged {n_logged} runs into MLflow experiment '{CASE_STUDY_ID}'")
```

### Query runs with `search_runs`

`search_runs` returns a DataFrame whose columns are prefixed by `params.` and `metrics.`,
reflecting how MLflow stores them: parameters are strings it does not interpret, metrics are
numbers it can order and filter by. That split is a modelling decision the caller makes at
logging time, and it decides what can be queried later.

```python
all_runs = mlflow.search_runs(
    experiment_names=[CASE_STUDY_ID],
    order_by=["metrics.ic_mean_daily DESC"],
)
print(f"Total runs tracked: {len(all_runs)}")
all_runs[
    [
        "run_id",
        "tags.mlflow.runName",
        "params.family",
        "params.label",
        "metrics.ic_mean_daily",
    ]
].head(10)
```

### Filter by family

`filter_string` takes SQL-like predicates over the same parameters and metrics, which is the
operation the grouped ranking in section 3 performed with a `GROUP BY`.

```python
gbm_runs = mlflow.search_runs(
    experiment_names=[CASE_STUDY_ID],
    filter_string="params.family = 'gbm'",
    order_by=["metrics.ic_mean_daily DESC"],
)
print(f"GBM runs: {len(gbm_runs)}")
gbm_runs[
    [
        "tags.mlflow.runName",
        "params.label",
        "params.config_name",
        "metrics.ic_mean_daily",
    ]
].head(10)
```

### Verify ranking parity with the manual catalog

The comparison is an outer join on family and label, so a group present in one system and
absent from the other fails rather than being dropped. Both the identity of each group's
top run and its metric are checked, because agreeing on the number while disagreeing on
which run produced it is the more interesting failure.

```python
mlflow_best = (
    all_runs.sort_values("metrics.ic_mean_daily", ascending=False)
    .groupby(["params.label", "params.family"], as_index=False)
    .first()
)
mlflow_best_compact = mlflow_best.rename(
    columns={
        "tags.mlflow.runName": "training_hash",
        "params.family": "family",
        "params.label": "label",
        "metrics.ic_mean_daily": "ic_mean",
    }
)[["training_hash", "family", "label", "ic_mean"]].sort_values(
    ["label", "ic_mean"], ascending=[True, False]
)

manual_best_compact = best_validation[["training_hash", "family", "label", "ic_mean"]].sort_values(
    ["label", "ic_mean"], ascending=[True, False]
)

comparison = mlflow_best_compact.merge(
    manual_best_compact,
    on=["family", "label"],
    how="outer",
    suffixes=("_mlflow", "_manual"),
    indicator=True,
)
comparison["ic_match"] = (
    comparison["ic_mean_mlflow"] - comparison["ic_mean_manual"]
).abs() < PARITY_TOLERANCE
comparison["hash_match"] = comparison["training_hash_mlflow"] == comparison["training_hash_manual"]
assert (comparison["_merge"] == "both").all()
assert comparison[["ic_match", "hash_match"]].all().all()
print("Ranking parity check (MLflow vs manual catalog):")
comparison
```

The assertions above are what make this a check: the notebook stops if any group is missing
from either side, or if the two disagree about which run tops it or by how much.

What that establishes is that MLflow's query returns the catalog it was given. It is a test
of the logging and of the query, and it says nothing about whether the metric is the right
one to rank on - which section 3 already answered, and the answer was no.

### Clean up

The tracking store was written under this chapter's output directory and is removed here. A
production store persists and is shared, which is most of what makes it a tracker rather
than a log file.

```python
shutil.rmtree(MLFLOW_DIR, ignore_errors=True)
print(f"Cleaned up tracking store: {MLFLOW_DIR}")
```

## Key Takeaways

1. An experiment tracker has to answer three questions - what was run, what came out, and
   can it be reproduced - and the third is the one a table of metrics cannot answer alone.
   A row that resolves to the specification and the artifacts is the difference between a
   record and a claim.
2. Identify a run by a hash of its specification. That makes the log idempotent, so
   re-running a pipeline cannot produce two rows describing the same experiment, and it
   makes a configuration change visible as a new identifier rather than as an edit.
3. Record the experiment budget, not only the results. A family tried a hundred ways and a
   family tried twice are not comparable on their best score, because the maximum of more
   draws is higher even when nothing improved. The count does not say how much higher; it
   makes the asymmetry visible, which a table of top runs per family does not.
4. Keep ranking separate from selection. Ordering the catalog by information coefficient
   says which model ranked the cross-section best; it is not what decides deployment, and a
   tracker that presents one ordering invites it to become the other.
5. The tool supplies the interface and the infrastructure; the discipline supplies the
   trust. Deterministic identifiers, a manifest that resolves and a recorded specification
   are properties of the pipeline that writes the runs, and MLflow reproduces this catalog
   exactly because they were already there.

**Known limitations**

- The parity check covers the runs that were logged, on one case study, with one metric. It
  establishes that the query returns the catalog it was given, not that either system is
  correctly configured for a catalog with a different shape.
- The tracking store here is local, single-user and deleted at the end. Multi-user
  collaboration, access control and a persistent web interface are most of what MLflow is
  chosen for, and none of them is exercised.
- The catalog reads two labels from one case study. A registry-wide view would also have to
  reconcile runs across case studies, which have their own registries.

**Next**: This is the final notebook of Chapter 26. The chapter prose closes with the
three-layer governance model - detection, response, automated safety - which together with
the MLOps components in these notebooks keep a deployed strategy auditable.
![notebook output](figures/p1_1.png)