Part 9 — Evaluate, Compare, and Save Your Fine-Tuned Model

Download the matching self-contained notebook · Read the notebook preview

What you will build

We now use the same source model and target splits to compare the unchanged base with five adaptation methods. The complete program includes every model class and utility needed for the comparison. No previous notebook execution is assumed.

The task is intentionally easy and small. If the unchanged base ties all adapted models, the selection rule may choose it. That is useful evidence that this particular exercise does not justify additional adaptation, rather than a result we should hide.

By the end: Compare adaptation methods, select with validation, and verify a saved model reload.

What you need before starting

You need basic Python: variables, functions, dictionaries, loops, and imports. When this lesson introduces a class or a library-specific term, its role is explained below.

Use Python 3.12. These examples were executed on Windows with Python 3.12.4. Use the shown interpreter consistently when installing packages and running code. This is an original small CPU teaching experiment, not a downloaded general-purpose LLM.

You do not need to run an earlier article to make this part execute. The complete files and data used here are supplied below. The series order is for learning the ideas progressively.

Understand the idea before the code

A metric is a number used to evaluate behavior. A hyperparameter is a setting you choose before a run, such as learning rate, rank, or training steps; it is different from a learned weight. Model selection means choosing a configuration using validation measurements. Generalization means working on relevant examples not used to fit that model.

For a particular class, true positives (TP) are examples correctly assigned to it, false positives (FP) are other classes incorrectly assigned to it, and false negatives (FN) are its examples assigned elsewhere. Precision asks “of predictions of this class, how many were right?” Recall asks “of the examples truly in this class, how many did we find?” Macro-F1 gives each class equal weight when averaging its F1 score.

A checkpoint in model training is a saved model/optimizer state or parameter artifact. Here we save a model state dictionary using state_dict(): a mapping of tensor names to stored values. This is not the same thing as a LangGraph workflow's ticket-state dictionary. Saving parameters also does not automatically save the Python architecture class or vocabulary.

Reload means construct the appropriate model structure and copy saved values into it. Reproducibility means recording enough code, data, settings, versions, and artifact details to repeat or inspect the experiment. It is broader than setting one random seed.

A picture of this part's idea

This picture introduces the mechanism. The numbers and assertions in the program below are the actual experiment.

Create the lesson folder

Create this folder and keep the filenames exactly as shown. In particular, a local module is an ordinary file beside the program, not an extra package you must guess how to install.

lesson-folder/
  lesson.py
  requirements.txt

Save requirements.txt

These are the external Python packages for this lesson. The local files above are included in the article; they are not pip packages.

torch==2.2.2
numpy==1.26.4
matplotlib==3.9.1

python -m venv .venv
.venv\Scripts\python.exe -m pip install -r requirements.txt

On Linux/macOS, use python3 -m venv .venv and .venv/bin/python -m pip install -r requirements.txt. Installation requires internet access. The default examples do not call a paid API.

Complete code for this article

Save this complete file as lesson.py

This is the complete program, including imports, data, function definitions, and the entry point.

import copy
import re
import torch
from torch import nn
from torch.nn import functional as F

LABELS = ["python", "linux", "iot"]

GROUPS = [
    ("python", ["python", "list", "dictionary", "function", "script", "package"]),
    ("linux", ["linux", "shell", "permission", "service", "process", "terminal"]),
    ("iot", ["esp32", "camera", "sensor", "gpio", "wifi", "raspberry"]),
]


def seed(value=42):
    torch.manual_seed(value)
    torch.set_num_threads(1)


def records():
    """One group per topic keyword; all paraphrases stay in the same split."""
    rows = []
    for label, words in GROUPS:
        for i, word in enumerate(words):
            split = "train" if i < 4 else "validation" if i == 4 else "test"
            for template in ["help with {}", "please explain {}", "debug my {}"]:
                rows.append(
                    dict(
                        text=template.format(word),
                        label=label,
                        group=f"{label}-{word}",
                        split=split,
                    )
                )
    return rows


def tokens(text):
    return re.findall("[a-z0-9]+", text.lower())


def make_vocab(rows):
    return {
        word: i
        for i, word in enumerate(sorted({w for r in rows for w in tokens(r["text"])}))
    }


def encode(rows, vocab):
    x = torch.zeros(len(rows), len(vocab))
    for i, row in enumerate(rows):
        for word in tokens(row["text"]):
            if word in vocab:
                x[i, vocab[word]] += 1
    return (x, torch.tensor([LABELS.index(r["label"]) for r in rows]))


class Router(nn.Module):
    def __init__(self, width, hidden=16):
        super().__init__()
        self.encoder = nn.Linear(width, hidden)
        self.head = nn.Linear(hidden, 3)

    def forward(self, x):
        return self.head(torch.tanh(self.encoder(x)))


class AdapterRouter(nn.Module):
    def __init__(self, base, rank=4):
        super().__init__()
        self.base = copy.deepcopy(base).requires_grad_(False)
        h = self.base.head.in_features
        self.down = nn.Linear(h, rank, bias=False)
        self.up = nn.Linear(rank, h, bias=False)
        nn.init.zeros_(self.up.weight)

    def forward(self, x):
        h = torch.tanh(self.base.encoder(x))
        return self.base.head(h + self.up(torch.relu(self.down(h))))


class LoRALinear(nn.Module):
    def __init__(self, base, rank=2, alpha=2):
        super().__init__()
        self.base = copy.deepcopy(base).requires_grad_(False)
        self.A = nn.Parameter(torch.randn(rank, base.in_features) * 0.02)
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))
        self.scale = alpha / rank

    def forward(self, x):
        return self.base(x) + x @ self.A.T @ self.B.T * self.scale

    def merged_weight(self):
        return self.base.weight + self.scale * self.B @ self.A


def train(model, x, y, steps=100, lr=0.03):
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.Adam(params, lr=lr)
    losses = []
    model.train()
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        loss = F.cross_entropy(model(x), y)
        loss.backward()
        optimizer.step()
        losses.append(float(loss.detach()))
    return losses


def prepare():
    """Pretrain on an explicitly separate, synthetic source corpus."""
    seed()
    target = records()
    source = []
    for label, words in GROUPS:
        for word in words:
            source.append(dict(text=f"general guide {word}", label=label))
    vocab = make_vocab(source + [r for r in target if r["split"] == "train"])
    sx, sy = encode(source, vocab)
    base = Router(len(vocab))
    train(base, sx, sy, steps=45, lr=0.02)
    return (base, vocab, target)


def method(base, name):
    model = copy.deepcopy(base)
    if name == "head":
        model.encoder.requires_grad_(False)
    elif name == "adapter":
        model = AdapterRouter(base)
    elif name == "lora":
        model.requires_grad_(False)
        model.encoder = LoRALinear(base.encoder, rank=2, alpha=2)
    elif name == "bitfit":
        for key, p in model.named_parameters():
            p.requires_grad_("bias" in key)
    elif name != "full":
        raise ValueError(name)
    return model


def counts(model):
    return {
        "trainable": sum((p.numel() for p in model.parameters() if p.requires_grad)),
        "total": sum((p.numel() for p in model.parameters())),
    }


def metrics(y_true, y_pred, size=3):
    matrix = torch.zeros(size, size, dtype=torch.int64)
    for truth, pred in zip(y_true, y_pred):
        matrix[int(truth), int(pred)] += 1
    f1 = []
    for i in range(size):
        tp = int(matrix[i, i])
        fp = int(matrix[:, i].sum()) - tp
        fn = int(matrix[i, :].sum()) - tp
        f1.append(2 * tp / max(1, 2 * tp + fp + fn))
    return {
        "accuracy": float(matrix.diag().sum() / max(1, int(matrix.sum()))),
        "macro_f1": sum(f1) / size,
        "confusion": matrix.tolist(),
    }


def main():
    import copy, json
    import torch

    base, vocab, rows = prepare()
    splits = {
        s: encode([r for r in rows if r["split"] == s], vocab)
        for s in ["train", "validation", "test"]
    }
    with torch.no_grad():
        base_prediction = base(splits["validation"][0]).argmax(-1)
    report = [
        {
            "method": "base",
            "trainable": 0,
            "total": counts(base)["total"],
            **metrics(splits["validation"][1], base_prediction),
        }
    ]
    models = {"base": copy.deepcopy(base)}
    for name in ["full", "head", "adapter", "bitfit", "lora"]:
        torch.manual_seed(42)
        model = method(base, name)
        train(model, *splits["train"], steps=80, lr=0.01)
        with torch.no_grad():
            prediction = model(splits["validation"][0]).argmax(-1)
        report.append(
            {
                "method": name,
                **counts(model),
                **metrics(splits["validation"][1], prediction),
            }
        )
        models[name] = model
    for row in report:
        print(row)
    chosen = min(report, key=lambda row: (-row["macro_f1"], row["trainable"]))["method"]
    model = models[chosen].eval()
    with torch.no_grad():
        pred = model(splits["test"][0]).argmax(-1)
    final = metrics(splits["test"][1], pred)
    print("selected:", chosen, "test:", final)
    import matplotlib.pyplot as plt

    fig, ax = plt.subplots(figsize=(4, 3))
    ax.imshow(final["confusion"], cmap="Blues")
    ax.set(
        xticks=range(3),
        yticks=range(3),
        xticklabels=LABELS,
        yticklabels=LABELS,
        xlabel="Predicted",
        ylabel="True",
        title="Synthetic held-out test",
    )
    for i in range(3):
        for j in range(3):
            ax.text(j, i, str(final["confusion"][i][j]), ha="center", va="center")
    plt.savefig("result-01.png", bbox_inches="tight")
    print("Saved result-01.png")
    plt.show()
    from pathlib import Path

    folder = Path("checkpoints")
    folder.mkdir(exist_ok=True)
    torch.save(model.state_dict(), folder / "router.pt")
    config = {
        "method": chosen,
        "vocab": vocab,
        "labels": LABELS,
        "seed": 42,
        "steps": 80,
        "learning_rate": 0.01,
        "torch": torch.__version__,
        "test": final,
    }
    (folder / "router.json").write_text(json.dumps(config, indent=2))
    restored = copy.deepcopy(base) if chosen == "base" else method(base, chosen)
    restored.load_state_dict(
        torch.load(folder / "router.pt", map_location="cpu", weights_only=True)
    )
    restored.eval()
    with torch.no_grad():
        assert torch.allclose(model(splits["test"][0]), restored(splits["test"][0]))
    print("Reloaded predictions match.")


if __name__ == "__main__":
    main()

Run it and compare the result

Run commands from the lesson folder. In the commands below, python means the environment you installed into. On Windows, substitute .venv\Scripts\python.exe; on Linux/macOS substitute .venv/bin/python. Do not create the environment with one interpreter and run the lesson with another.

Run 1: python lesson.py

python lesson.py

Captured standard output:

{'method': 'base', 'trainable': 0, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'full', 'trainable': 483, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'head', 'trainable': 51, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'adapter', 'trainable': 128, 'total': 611, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'bitfit', 'trainable': 19, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'lora', 'trainable': 84, 'total': 567, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
selected: base test: {'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
Saved result-01.png
Reloaded predictions match.

The program saves result-01.png in the working folder. This is output from the shown code, not a separately invented chart.

The displayed output was captured from the supplied default program in a separate folder, with no project-level module on its import path. It is a reproducible demonstration on these examples, not a claim about arbitrary inputs.

Walk through what just happened

1. Control the comparison

Each method starts from the same base. The loop records trainable parameter count and validation metrics using the same data and declared update budget.

2. Inspect errors by class

metrics forms a confusion matrix and computes accuracy and macro-F1. The saved plot places true classes on rows and predicted classes on columns.

3. Select before final testing

The example chooses by validation macro-F1 and breaks ties by trainable count. It evaluates only that choice on the test split. This tie-break is an exercise rule, not a universal deployment policy.

4. Save enough to reload

The program writes tensor values and a JSON configuration containing vocabulary, label order, method, and settings. It rebuilds the matching structure, reloads weights, and checks identical predictions.

Change one thing yourself

Add an independent challenge set containing ambiguous or unsupported topics. Keep it separate from the original test score and explain what new behavior it measures.

If your result is different

A state_dict stores tensors, not the Python class definition or the vocabulary automatically. Changing label order after loading changes the interpretation of output indices.

Check the complete imports and filenames first. Then compare your input and command with the shown run. Numeric results may differ slightly between platforms; the program's assertions check the intended relationships rather than relying on a decorative screenshot.

What you can now explain

Compare adaptation methods, select with validation, and verify a saved model reload. Next we will build on this result in Part 10.

Technical reference: PyTorch learning basics. This project and its wording are original CodeTips Academy material.