Compare five adaptation methods, inspect a confusion matrix, and verify a saved model reloads correctly.
Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.
We now compare full tuning, head-only training, a bottleneck adapter, bias-only training, and LoRA on the same miniature routing problem. Each method receives a copy of the same pretrained base. We measure validation accuracy, macro-F1, and trainable parameter count. These are controlled teaching comparisons, not rankings of methods for large language models.
The dataset is tiny and synthetic. If every method reaches the same score, the correct conclusion is that this task does not distinguish them well. Do not add hidden examples or change the seed until your favorite method wins. Preserve the result and explain its limits.
Accuracy is the fraction of predictions that are correct. If 7 of 9 questions are correct, accuracy is 77.8%. A confusion matrix puts true labels on rows and predicted labels on columns. A count at row iot, column linux is an IoT request misrouted to Linux.
For one class, precision is TP/(TP+FP) and recall is TP/(TP+FN). F1 combines them as 2*TP/(2*TP+FP+FN). Macro-F1 averages class F1 values equally. This is useful when a common class could dominate accuracy, although no single metric explains all errors.
import copy, json
import torch
from ft_lab import prepare, method, encode, train, counts, metrics, LABELS
base,vocab,rows=prepare()
splits={s:encode([r for r in rows if r['split']==s],vocab) for s in ['train','validation','test']}
with torch.no_grad(): base_prediction=base(splits['validation'][0]).argmax(-1)
report=[{'method':'base', 'trainable':0, 'total':counts(base)['total'],
**metrics(splits['validation'][1],base_prediction)}]
models={'base':copy.deepcopy(base)}
for name in ['full','head','adapter','bitfit','lora']:
torch.manual_seed(42)
model=method(base,name)
train(model,*splits['train'],steps=80,lr=.01)
with torch.no_grad(): prediction=model(splits['validation'][0]).argmax(-1)
report.append({'method':name,**counts(model),**metrics(splits['validation'][1],prediction)})
models[name]=model
for row in report: print(row)Executed output (CPU teaching experiment):
{'method': 'base', 'trainable': 0, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'full', 'trainable': 483, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'head', 'trainable': 51, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'adapter', 'trainable': 128, 'total': 611, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'bitfit', 'trainable': 19, 'total': 483, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
{'method': 'lora', 'trainable': 84, 'total': 567, 'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
The unchanged pretrained base is included with zero adaptation parameters. It may already solve this simple task; selecting it is a useful result, because training would add cost without measured benefit.
For this demonstration, select the highest validation macro-F1 and break ties by trainable parameter count. This tie-break is an explicit exercise rule, not a universal production recommendation. Latency, licensing, memory, and maintainability could change your real decision.
We then evaluate only the selected model on the test split. Remember the source corpus already contained the keywords; this test measures new target phrasing under the declared source knowledge. Nine examples give a very uncertain estimate. One extra wrong prediction changes accuracy by 11.1 percentage points.
chosen=min(report,key=lambda row:(-row['macro_f1'],row['trainable']))['method']
model=models[chosen].eval()
with torch.no_grad(): pred=model(splits['test'][0]).argmax(-1)
final=metrics(splits['test'][1],pred)
print('selected:',chosen,'test:',final)
import matplotlib.pyplot as plt
fig,ax=plt.subplots(figsize=(4,3))
ax.imshow(final['confusion'],cmap='Blues')
ax.set(xticks=range(3),yticks=range(3),xticklabels=LABELS,yticklabels=LABELS,
xlabel='Predicted',ylabel='True',title='Synthetic held-out test')
for i in range(3):
for j in range(3): ax.text(j,i,str(final['confusion'][i][j]),ha='center',va='center')
plt.show()Executed output (CPU teaching experiment):
selected: base test: {'accuracy': 1.0, 'macro_f1': 1.0, 'confusion': [[3, 0, 0], [0, 3, 0], [0, 0, 3]]}
Save the architecture choice, vocabulary, label order, training settings, dependency versions, and model tensors. If the tokenizer changes, the same input can become different numbers. If the label order changes, a correct class index can be displayed with the wrong name.
The example below stores a full copy of the selected small model's state for simplicity, including its frozen base when applicable. It is not an adapter-only export. Production LoRA packaging normally stores adapter configuration and weights together with an exact base-model reference. Only load model files from a source you trust.
from pathlib import Path
folder=Path('checkpoints');folder.mkdir(exist_ok=True)
torch.save(model.state_dict(),folder/'router.pt')
config={'method':chosen,'vocab':vocab,'labels':LABELS,'seed':42,
'steps':80,'learning_rate':.01,'torch':torch.__version__,'test':final}
(folder/'router.json').write_text(json.dumps(config,indent=2))
restored=copy.deepcopy(base) if chosen=='base' else method(base,chosen)
restored.load_state_dict(torch.load(folder/'router.pt',map_location='cpu',weights_only=True))
restored.eval()
with torch.no_grad():
assert torch.allclose(model(splits['test'][0]),restored(splits['test'][0]))
print('Reloaded predictions match.')Executed output (CPU teaching experiment):
Reloaded predictions match.
Inspect every error in a small test. Add separate challenge sets for ambiguous wording, misspellings, mixed Python/IoT questions, and unsupported topics. Do not mix those exploratory sets into the final score without explaining the new evaluation definition. For generated responses, validate JSON structure and review factual claims instead of reporting classifier accuracy.
Measure the unchanged base too. Check old tasks for forgetting. Record peak memory and latency on the machine where you plan to run the model. Trainable parameter count is not a direct memory measurement and not a latency prediction.
Your final model card should say what was trained, on what data, with which split, which base, which versions, and what is not established. Our release report distinguishes executed CPU notebooks from optional external-model training. Next, we will connect the concepts to a Hugging Face pretrained model and show how to publish the notebooks with stable links.