CODETIPS ACADEMY / FINE-TUNING METHODS

Part 5 — LoRA from Scratch: Two Small Matrices Instead of One Large Update

Calculate a LoRA update by hand, train it, and prove that merging preserves predictions.

Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.

The update is what we compress

LoRA means Low-Rank Adaptation. Keep an existing weight matrix W frozen. Learn an update represented by two smaller matrices: delta_W = (alpha/r) * B @ A. The adapted transformation is W_new = W + delta_W. The base remains useful; the task-specific change is stored separately.

If W has shape [output_width, input_width], choose A with shape [rank, input_width] and B with shape [output_width, rank]. Rank, written r, limits the number of independent directions in the update. It is a capacity choice, not a percentage of training examples.

A numerical example you can verify

Let W = [[1,0],[0,1]], the identity matrix. Let rank be 1, A = [[1,2]], and B = [[0.1],[0.2]]. With alpha/r = 1, the update is [[0.1,0.2],[0.2,0.4]]. For input [2,1], the base output is [2,1]. The update adds [0.4,0.8], so the adapted output is [2.4,1.8].

PyTorch's linear operation uses a row input multiplied by W.T. That transpose is easy to miss when translating the formula into code. The cell below calculates both paths and verifies they match.

import torch
W=torch.eye(2)
A=torch.tensor([[1.,2.]])
B=torch.tensor([[.1],[.2]])
x=torch.tensor([[2.,1.]])
separate=x@W.T + x@A.T@B.T
merged=x@(W+B@A).T
print('delta W:',B@A,'output:',separate)
assert torch.allclose(separate,torch.tensor([[2.4,1.8]]))
assert torch.allclose(separate,merged)

delta W: tensor([[0.1000, 0.2000],
        [0.2000, 0.4000]]) output: tensor([[2.4000, 1.8000]])

This tiny example does not save parameters: two rank-one matrices contain four numbers, just like a 2-by-2 matrix. Savings appear when the dimensions are large relative to the rank. A 4096-by-4096 matrix has 16,777,216 numbers. At rank 8, the LoRA matrices have 8*4096 + 4096*8 = 65,536 numbers: 1/256 as many for this matrix update. This count excludes the frozen base, other layers, gradients, optimizer state, and activations.

Train a real LoRA update in our router

LoRALinear stores a frozen copy of an existing layer, randomly initializes A, and initializes B to zero. The starting delta is zero, so the wrapper initially preserves the base function. If both A and B were zero, both would initially receive zero gradient in this multiplication. One random factor avoids that dead start.

from ft_lab import prepare, method, encode, train, counts
base,vocab,rows=prepare()
x,y=encode([r for r in rows if r['split']=='train'],vocab)
model=method(base,'lora')
assert torch.allclose(model(x),base(x))
frozen=model.encoder.base.weight.detach().clone()
history=train(model,x,y,steps=100,lr=.02)
assert torch.equal(frozen,model.encoder.base.weight)
with torch.no_grad():
    separate=model.encoder(x)
    merged=torch.nn.functional.linear(x,model.encoder.merged_weight(),model.encoder.base.bias)
assert torch.allclose(separate,merged,atol=1e-5)
print(counts(model),'merge max error:',float((separate-merged).abs().max()))

{'trainable': 84, 'total': 567} merge max error: 2.384185791015625e-07

We attach LoRA only to the router encoder and freeze the head. In transformer fine-tuning, you choose target modules such as attention projections or all suitable linear layers. The exact names depend on the architecture. Do not blindly copy q_proj into a model that has different module names.

alpha/r scales the learned update in this implementation. Other variants use different scaling. Increasing rank changes both the number of parameters and the scaling when alpha is unchanged, so “higher rank” is not an isolated experiment unless you track those settings.

Saving and merging

An adapter checkpoint can be small because it stores only the added tensors and configuration. It still depends on the exact base model, tokenizer, target layers, rank, and scaling. A LoRA file by itself is not a complete chatbot.

For ordinary unquantized linear layers, merging adds the update into the base matrix. Our assertion compares the two calculations before and after merging. Small floating-point differences are normal. Quantized deployment can require different handling; do not assume a method that merges an ordinary float matrix safely merges every quantized implementation.

What LoRA does and does not guarantee

Fewer trainable parameters usually reduce adapter storage and optimizer state. They do not eliminate the frozen model or the forward/backward activation costs. Quality depends on the data, rank, target layers, and optimization. On our tiny task, LoRA may tie a head-only method; that does not invalidate either mechanism.

Try rank 1 versus 4 while preserving the same starting model. Inspect training loss and validation errors, then record the configuration. Next, we will reduce the precision used to store the frozen base and examine where QLoRA fits.

Method reference: the original LoRA paper. Implementation reference: PEFT's LoRA guide.

← Part 4 · Part 6 →