כוונון עדין של שנאי סנטימנט פיננסי ללא דליפה ממערך הבחינה

# Fine-tuning a transformer, and checking what its checkpoint already saw


# Fine-tuning a transformer, and checking what its checkpoint already saw

**Chapter 10: text feature engineering**

**Docker image**: `ml4t-gpu`

**Section reference**: Sections 10.4 and 10.5

## What this notebook is for

Fine-tuning a pre-trained transformer for sentence classification is a short and
well-supported piece of work: load a checkpoint, tokenize, hand it to the Trainer, read
the metrics. This notebook does that three times, on one domain-specific checkpoint and
two general ones, so the mechanics are visible on a task small enough to run.

The comparison it produces is the part worth being careful about. One of the three
checkpoints, `ProsusAI/finbert`, was itself fine-tuned on the Financial PhraseBank - the
corpus this notebook's test split is drawn from. Its test score is therefore measured on
sentences it has already been trained on, and it is not a held-out number. That fact is
stated where the models are introduced and carried in the results table, because a
comparison table is read long before any caveat at the end of a notebook.

## Learning objectives

After working through this notebook you will be able to:

- Fine-tune a pre-trained transformer for sentence classification with the Hugging Face
  Trainer, and say what each of the training arguments changes.
- Check whether a published checkpoint was trained on the data you are about to test it
  on, and say what that does to the number you get.
- Read a confusion matrix for a three-class sentiment task and identify which pair of
  classes a model actually confuses.
- Decide whether a difference between two models on one test split is large enough to act
  on.

## Prerequisites

- Sections 10.4 and 10.5 of the chapter.
- `03_sentiment_evolution` for the baselines these models are being compared against.

## Related notebooks

- `03_sentiment_evolution.py` - lexicon, TF-IDF and static-embedding baselines
- `06_finbert_cross_dataset.py` - the published FinBERT checkpoint scored zero-shot on a
  different corpus
- `12_gradient_boosting/10_shap_nlp_sentiment.py` - attributing a text model's decisions

## What it costs to run

Three fine-tuning runs on one GPU. The training time each takes is measured and reported
below rather than asserted here, because it depends on the card.

```python
"""Fine-tune three transformer checkpoints on financial sentiment and compare them."""

import json
import time
import warnings
from functools import partial

import matplotlib.pyplot as plt
import numpy as np
import polars as pl
import seaborn as sns
import torch

# `multiprocess`, reached through these two, raises a SyntaxWarning at COMPILE time, so a
# module-level filter set after the import is too late and the warning reached the committed
# render carrying the absolute path of whichever checkout produced it.
with warnings.catch_warnings():
    warnings.simplefilter("ignore", SyntaxWarning)
    import evaluate
    from datasets import Dataset, DatasetDict

from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    DataCollatorWithPadding,
    EarlyStoppingCallback,
    Trainer,
    TrainingArguments,
)
from transformers import (
    set_seed as set_transformers_seed,
)
from transformers import utils as transformers_utils

transformers_logging = transformers_utils.logging

from data import load_financial_phrasebank
from utils.paths import get_chapter_dir
from utils.reproducibility import set_global_seeds
from utils.style import COLORS, FIGSIZE, show_with_alt

# The sentencepiece-to-fast conversion warns about byte fallback once per tokenizer built
# that way. It reports a property of the conversion, not of this notebook's data.
warnings.filterwarnings(
    "ignore", category=UserWarning, module="transformers.convert_slow_tokenizer"
)
```

The two parameters below are what a reduced run overrides. `MAX_TRAIN_STEPS` caps the total
number of optimizer steps; at -1 the Trainer runs the configured epochs instead. Neither
declaration carries a trailing comment, because papermill parses this cell line by line and
a comment on the same line hides the name from it - the override is then dropped in silence
and the reduced run trains at production size.

```python
SEED = 42
MAX_TRAIN_STEPS = -1
# How many sentences to score in the validation and test splits; 0 means all of them.
# This is the knob that decides whether CI can run this notebook - see the comment above
# the split below.
MAX_EVAL_SAMPLES = 0
```

`set_global_seeds` covers Python, NumPy and Torch. The Trainer draws from its own generator
for shuffling and dropout, which needs seeding separately or the run is not reproducible
even with everything else pinned.

```python
set_global_seeds(SEED)
set_transformers_seed(SEED)

CONFIG = {
    "random_seed": SEED,
    "dataset": {
        "name": "takala/financial_phrasebank",
        "subset": "sentences_allagree",
        "test_size": 0.15,
        "val_size": 0.15,
    },
    # `saw_phrasebank` is the field that decides how each row of the results table may be
    # read. It is a property of the published checkpoint, not of anything this notebook
    # does, and it is declared here so no comparison below can quietly omit it.
    "models": {
        "finbert": {
            "model_id": "ProsusAI/finbert",
            "description": "BERT already fine-tuned for sentiment on Financial PhraseBank",
            "saw_phrasebank": True,
        },
        "deberta": {
            "model_id": "microsoft/deberta-v3-small",
            "description": "General checkpoint, disentangled attention",
            "saw_phrasebank": False,
        },
        "modernbert": {
            "model_id": "answerdotai/ModernBERT-base",
            "description": "General checkpoint, long context window",
            "saw_phrasebank": False,
        },
    },
    "training": {
        "learning_rate": 2e-5,
        "batch_size": 16,
        "num_epochs": 3,
        "weight_decay": 0.01,
        "warmup_ratio": 0.1,
        "max_length": 128,
        "early_stopping_patience": 2,
    },
}

print(json.dumps(CONFIG, indent=2))

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")
```

## The corpus and the split

The `sentences_allagree` subset keeps only sentences every annotator scored the same way.
The split is stratified on the label, because the three classes are far from balanced and
an unstratified draw would leave the test set with a different class mix from the training
set, which moves accuracy for reasons that have nothing to do with the model.

```python
df = load_financial_phrasebank(agreement="100")
print(f"Dataset size: {len(df):,}")
```

```python
dataset_config = CONFIG["dataset"]
held_out_fraction = dataset_config["test_size"] + dataset_config["val_size"]

df_pd = df.to_pandas()
train_pd, temp_pd = train_test_split(
    df_pd, test_size=held_out_fraction, random_state=SEED, stratify=df_pd["label"]
)
val_pd, test_pd = train_test_split(
    temp_pd,
    test_size=dataset_config["test_size"] / held_out_fraction,
    random_state=SEED,
    stratify=temp_pd["label"],
)

train_df = pl.from_pandas(train_pd)
val_df = pl.from_pandas(val_pd)
test_df = pl.from_pandas(test_pd)

print(f"Train: {len(train_df)}, Val: {len(val_df)}, Test: {len(test_df)}")
```

The Trainer reads a `DatasetDict`, which is built from pandas. Polars has no direct
conversion into it, so pandas is the boundary here rather than a preference.

```python
def create_dataset_dict(train_df: pl.DataFrame, val_df: pl.DataFrame, test_df: pl.DataFrame):
    return DatasetDict(
        {
            "train": Dataset.from_pandas(
                train_df.select(["sentence", "label"]).to_pandas(), preserve_index=False
            ),
            "validation": Dataset.from_pandas(
                val_df.select(["sentence", "label"]).to_pandas(), preserve_index=False
            ),
            "test": Dataset.from_pandas(
                test_df.select(["sentence", "label"]).to_pandas(), preserve_index=False
            ),
        }
    )


dataset = create_dataset_dict(train_df, val_df, test_df)
```

Scoring, not training, is what this notebook spends its time on, and `MAX_TRAIN_STEPS` does
not touch it. Each model is scored on the validation split every `eval_steps` during training
and once on the test split at the end, so what the runtime tracks is the number of scored
sentences, not the number of training steps. Bounding the scored splits is the knob that
reaches that; bounding the model count would leave a three-way comparison with one entry.

`MAX_EVAL_SAMPLES = 0` means score everything, which is what a real run does. The splits are
already shuffled and stratified by `train_test_split`, so a prefix of each is class-balanced
in expectation.

```python
if MAX_EVAL_SAMPLES > 0:
    dataset = DatasetDict(
        {
            "train": dataset["train"],
            "validation": dataset["validation"].select(
                range(min(MAX_EVAL_SAMPLES, len(dataset["validation"])))
            ),
            "test": dataset["test"].select(range(min(MAX_EVAL_SAMPLES, len(dataset["test"])))),
        }
    )
    print(
        f"Scored splits bounded to {MAX_EVAL_SAMPLES}: "
        f"val {len(dataset['validation'])}, test {len(dataset['test'])}"
    )
```

## The three checkpoints, and which of them can be read as held out

`ProsusAI/finbert` is a BERT that its authors already fine-tuned for sentiment on the
Financial PhraseBank. The other two are general checkpoints that have not seen it. All
three are fine-tuned here on the same training split, so the mechanics are identical, but
only two of the three produce a test score with the usual meaning.

For FinBERT the test sentences are not unseen: they were in the corpus its published
weights were trained on. Whatever it scores here is an upper bound inflated by that
exposure, and it cannot be compared with the other two or attributed to the fine-tuning
step this notebook performs. It is kept in the comparison because a reader will reach for
a domain checkpoint first and should see what checking its provenance is worth.

```python
MODELS = {
    "FinBERT": CONFIG["models"]["finbert"],
    "DeBERTa-v3": CONFIG["models"]["deberta"],
    "ModernBERT": CONFIG["models"]["modernbert"],
}

id2label = {0: "negative", 1: "neutral", 2: "positive"}
label2id = {"negative": 0, "neutral": 1, "positive": 2}
```

Tokenization does not pad. Padding every sentence to `max_length` would spend most of the
compute on padding tokens, because these sentences are far shorter than the limit;
`DataCollatorWithPadding` instead pads each batch to its own longest member.

```python
def tokenize_function(examples, tokenizer, max_length=128):
    return tokenizer(
        examples["sentence"],
        truncation=True,
        max_length=max_length,
        # Note: Don't pad here; use DataCollatorWithPadding for dynamic padding
    )
```

Two metrics, and the second is the one to read. Accuracy on a corpus this imbalanced is
dominated by the majority class; macro F1 averages the per-class scores with equal weight,
so a model that never predicts the smallest class cannot hide behind the other two.

```python
accuracy_metric = evaluate.load("accuracy")
f1_metric = evaluate.load("f1")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    acc = accuracy_metric.compute(predictions=predictions, references=labels)
    f1 = f1_metric.compute(predictions=predictions, references=labels, average="macro")
    return {"accuracy": acc["accuracy"], "f1": f1["f1"]}
```

```python
def fine_tune_model(model_name: str, spec: dict, dataset: DatasetDict) -> dict:
    """Fine-tune one checkpoint for sentiment classification and score it on the test split."""
    model_path = spec["model_id"]
    print(f"Fine-tuning {model_name} from {model_path}")

    tokenizer = AutoTokenizer.from_pretrained(model_path)

    # Handle models without pad token (proper approach for encoder-only models)
    # Encoder models like BERT don't have eos_token; use [PAD] or add one
    if tokenizer.pad_token is None:
        if tokenizer.eos_token is not None:
            tokenizer.pad_token = tokenizer.eos_token
        else:
            # For BERT-like models, add [PAD] token
            tokenizer.add_special_tokens({"pad_token": "[PAD]"})

    model = AutoModelForSequenceClassification.from_pretrained(
        model_path,
        num_labels=3,
        id2label=id2label,
        label2id=label2id,
        ignore_mismatched_sizes=True,
    )

    # Resize embeddings if we added a new token
    model.resize_token_embeddings(len(tokenizer))

    # Handle pad token in model config
    if model.config.pad_token_id is None:
        model.config.pad_token_id = tokenizer.pad_token_id

    # Create data collator for dynamic padding (more efficient than max_length padding)
    data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

    # Tokenize dataset
    # `partial`, not a lambda: `datasets` pickles the transform to fingerprint it, and a
    # lambda defined here cannot be pickled, so the cache never hits.
    tokenized = dataset.map(
        partial(tokenize_function, tokenizer=tokenizer),
        batched=True,
        remove_columns=["sentence"],
    )

    # Training arguments - save checkpoints under chapter output directory
    chapter_dir = get_chapter_dir(10)
    output_dir = chapter_dir / "output" / "bert_finetuning" / model_name.lower().replace("-", "_")
    output_dir.mkdir(parents=True, exist_ok=True)

    # Use CONFIG values consistently
    train_config = CONFIG["training"]
    num_epochs = train_config["num_epochs"]
    batch_size = train_config["batch_size"]
    train_size = len(tokenized["train"])

    max_steps = MAX_TRAIN_STEPS

    # `load_best_model_at_end` requires save_steps to be a multiple of eval_steps, so the two
    # are derived together rather than set independently.
    if max_steps > 0:
        eval_steps = max(10, max_steps // 5)
        # save_steps must be a multiple of eval_steps
        save_steps = eval_steps * 2  # evaluate twice, save once
    else:
        eval_steps = None
        save_steps = None

    training_kwargs = {
        "output_dir": str(output_dir),
        "num_train_epochs": num_epochs,
        "max_steps": max_steps,
        "per_device_train_batch_size": batch_size,
        "per_device_eval_batch_size": batch_size * 2,
        "warmup_steps": min(50, max(10, train_size // batch_size // 4)),
        "weight_decay": train_config["weight_decay"],
        "logging_steps": max(10, train_size // batch_size // 3),
        "eval_strategy": "steps" if max_steps > 0 else "epoch",
        "eval_steps": eval_steps,
        "save_strategy": "steps" if max_steps > 0 else "epoch",
        "save_steps": save_steps,
        "load_best_model_at_end": True,
        "metric_for_best_model": "f1",
        "greater_is_better": True,
        "report_to": "none",  # Disable wandb/tensorboard
        "fp16": torch.cuda.is_available(),
    }

    training_args = TrainingArguments(**training_kwargs)

    # Create trainer with data collator for dynamic padding
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized["train"],
        eval_dataset=tokenized["validation"],
        data_collator=data_collator,
        compute_metrics=compute_metrics,
        callbacks=[
            EarlyStoppingCallback(early_stopping_patience=train_config["early_stopping_patience"])
        ],
    )

    # Train
    start_time = time.time()
    train_result = trainer.train()
    train_time = time.time() - start_time

    # One forward pass over the test split, not two. `predict` computes the same metrics
    # `evaluate` does when `compute_metrics` is set, and `metric_key_prefix="eval"` keeps
    # the keys the caller reads below.
    predictions = trainer.predict(tokenized["test"], metric_key_prefix="eval")
    test_results = predictions.metrics
    y_pred = np.argmax(predictions.predictions, axis=-1)
    y_true = predictions.label_ids

    return {
        "model_name": model_name,
        "accuracy": test_results["eval_accuracy"],
        "f1": test_results["eval_f1"],
        "train_time": train_time,
        "num_params": sum(p.numel() for p in model.parameters()),
        "saw_phrasebank": spec["saw_phrasebank"],
        "y_pred": y_pred,
        "y_true": y_true,
        "train_loss": train_result.training_loss,
    }
```

## Fine-tuning the three

What `transformers` reports at load is worth reading before the log is quieted, and it does
not say the same thing about all three.

DeBERTa-v3 and ModernBERT are general checkpoints with no classification head for any task,
so the library builds one with random weights and says so: "some weights were newly
initialized ... you should probably TRAIN this model". That is fine-tuning in one sentence -
the encoder arrives trained, the head does not, and what follows trains the head while
adjusting the encoder underneath it.

FinBERT is not in that message, because it already carries a trained three-class sentiment
head and the shapes match, so those weights are kept. That is not the advantage it sounds
like. Its head was trained under its own label order, `0: positive, 1: negative, 2:
neutral`, and this notebook assigns `0: negative, 1: neutral, 2: positive` to match the
corpus. Keeping the weights while relabeling the outputs leaves a head whose learned
meanings are permuted against the labels it is now scored on, so fine-tuning has to undo
the permutation before it can improve on anything.

Which sharpens what the leakage is. What FinBERT brings here is not a ready-made head - it
starts with a misaligned one - it is an encoder that has already read the sentences this
notebook is about to test it on.

```python
transformers_logging.set_verbosity_error()
```

```python
results = {name: fine_tune_model(name, spec, dataset) for name, spec in MODELS.items()}
```

## The comparison, with the contaminated row marked

The last column is what makes the table readable. A row whose checkpoint had already been
trained on this corpus is not a held-out measurement, and ranking it against the rows that
are compares two different things.

```python
summary_df = pl.DataFrame(
    [
        {
            "Model": r["model_name"],
            "Accuracy": f"{r['accuracy']:.1%}",
            "F1 (macro)": f"{r['f1']:.3f}",
            "Parameters": f"{r['num_params'] / 1e6:.1f}M",
            "Train Time": f"{r['train_time']:.0f}s",
            "Test split held out?": "no, checkpoint saw it" if r["saw_phrasebank"] else "yes",
        }
        for r in results.values()
    ]
)

summary_df
```

Two panels, and neither of them ranks the models. Read the left one for how little
separates the three scores against the range the axis could take, and remember that the
asterisked model is not measured on held-out data. Read the right one against it: the
training costs differ from one another far more than the scores do, which is the practical
finding here. The right panel's times are for whatever device this run used, printed above.

```python
fig, axes = plt.subplots(1, 2, figsize=FIGSIZE["dual_h_tall"])

ax = axes[0]
model_names = list(results.keys())
accuracies = [results[m]["accuracy"] for m in model_names]
f1_scores = [results[m]["f1"] for m in model_names]
bar_labels = [f"{m}*" if results[m]["saw_phrasebank"] else m for m in model_names]

x = np.arange(len(model_names))
width = 0.35

ax.bar(x - width / 2, accuracies, width, label="Accuracy", color=COLORS["blue"])
ax.bar(x + width / 2, f1_scores, width, label="F1 (macro)", color=COLORS["amber"])

ax.set_ylabel("Score")
ax.set_title("Test accuracy and macro F1 by model")
ax.set_xticks(x)
ax.set_xticklabels(bar_labels, fontsize=7)
ax.set_ylim(0, 1.18)
ax.legend(fontsize=7, loc="upper right", framealpha=0.9)

ax = axes[1]
times = [results[m]["train_time"] for m in model_names]
bars = ax.bar(model_names, times, color=COLORS["slate"])
ax.set_ylabel("Seconds")
ax.set_title("Fine-tuning wall-clock time by model")
ax.tick_params(axis="x", labelsize=7)

for bar in bars:
    height = bar.get_height()
    ax.annotate(
        f"{height:.0f}s",
        xy=(bar.get_x() + bar.get_width() / 2, height),
        xytext=(0, 3),
        textcoords="offset points",
        ha="center",
        va="bottom",
        fontsize=7,
    )

show_with_alt(
    fig,
    "Two panels. The left has a pair of bars per model, accuracy and macro F1, on an axis "
    "running from zero to one; all six bars are tall and close to the same height, and the "
    "model whose checkpoint had already seen the test corpus is marked with an asterisk. "
    "The right has one bar per model for training time in seconds, each labeled with its "
    "value, and the three differ from one another far more than the scores on the left do.",
)
```

A single accuracy figure says how often a model is right, not what it is wrong about. The
matrices below say which pair of classes each model confuses, which is the difference
between a model that is unsure and one that has collapsed a class. Worth noting is whether
the large off-diagonal cells fall in the same places across the three panels. That is an
observation about these three runs, not a measurement of the task: correlated errors are
equally consistent with a shared limitation in the training split or with the class
imbalance, and separating those would mean reading the misclassified sentences and their
labels rather than the matrices.

```python
n_models = len(results)
# `squeeze=False` so this cell does not depend on how many models ran. `n_models` is
# read from `results`, and without it a one-model dict gets a bare Axes rather than an
# array and the zip below raises `'Axes' object is not iterable`.
fig, axes = plt.subplots(1, n_models, figsize=FIGSIZE["triple_h_tall"], squeeze=False)

labels = ["negative", "neutral", "positive"]

for ax, (name, r) in zip(axes[0], results.items(), strict=True):
    cm = confusion_matrix(r["y_true"], r["y_pred"])
    sns.heatmap(
        cm,
        annot=True,
        fmt="d",
        cmap="Blues",
        cbar=False,
        annot_kws={"size": 7},
        xticklabels=labels,
        yticklabels=labels,
        ax=ax,
    )
    # Each model's score is in the table above, not repeated in the title, so a re-run does
    # not leave a figure asserting a number the table has already moved.
    ax.set_title(name, fontsize=8)
    ax.set_xlabel("Predicted")
    ax.set_ylabel("Actual")
    ax.tick_params(labelsize=7)

fig.suptitle("Test-set confusion matrices by model")
show_with_alt(
    fig,
    "Three heatmaps side by side, one per model, each a three-by-three grid of counts with "
    "the true class down the side and the predicted class across the bottom. In all three the "
    "diagonal cells carry far larger counts than any cell off it. Two of the panels have "
    "their largest off-diagonal count in the same cell, true neutral predicted positive; the "
    "third has no cell that stands out from the other off-diagonal counts.",
)
```

## Key takeaways

1. **Check what a published checkpoint was trained on before you test it.** `ProsusAI/finbert`
   was fine-tuned on the whole Financial PhraseBank, so a test split drawn from that corpus
   measures it on sentences it has already seen. Nothing in the training code, the metrics
   or the confusion matrix reveals this; it is on the model card, and reading the card is
   the step. A domain checkpoint is the first thing anyone reaches for, which is exactly why
   this trap is common.
2. **A leaked comparison does not announce itself as one.** A contaminated row does not have
   to be the strongest row, so scanning a results table for a suspiciously good score will
   not find it. What marks it is provenance, and the table has to carry that as a column
   rather than as a caveat at the end of the notebook, because the table is what gets read
   and quoted.
3. **Cross-notebook before-and-after comparisons need the same checkpoint on both sides.**
   `03_sentiment_evolution` scores `yiyanghkust/finbert-tone`, trained on analyst reports.
   Reading its number against this notebook's `ProsusAI/finbert` compares two different
   models, not one model before and after fine-tuning.
4. **Accuracy on an imbalanced corpus flatters the majority class.** Macro F1 weights the
   three classes equally, and the confusion matrix says which pair a model actually confuses.
   Read all three; a single scalar cannot distinguish a model that is unsure from one that
   has stopped predicting a class.
5. **Fine-tuning cost and fine-tuning benefit are not on the same scale here.** Set the
   spread of the training times against the spread of the scores before choosing between
   these checkpoints: on a task this size that choice is closer to an engineering decision
   than a modelling one.
6. **A confusion matrix says what a model got wrong, not why.** It locates a disagreement
   between two classes and cannot attribute it, and two models landing on the same pair does
   not change that. Whether the pair is genuinely ambiguous, under-represented in the training
   split, or inconsistently labeled is a question about the examples, and it is answered by
   reading them.

### The scope these numbers have

One stratified split of one corpus, at one seed. Transfer to text from another source is
measured in `06_finbert_cross_dataset`, and whether any of this carries a tradable signal in
`07_news_return_signals` and `09_filing_text_signals`. A difference of a point or two
between two models on a single split of a few hundred test sentences is inside the range a
different seed moves them.

```python
for r in results.values():
    held_out = "held-out" if not r["saw_phrasebank"] else "NOT held out, checkpoint saw this corpus"
    print(f"{r['model_name']}: accuracy {r['accuracy']:.1%}, macro F1 {r['f1']:.3f} ({held_out})")
    print(f"  {r['num_params'] / 1e6:.1f}M parameters, {r['train_time']:.0f}s to fine-tune")
```
![notebook output](figures/p1_1.png)
![notebook output](figures/p1_2.png)