Part 8 — Preference Tuning and DPO: Learn Which Answer Is Better

Download the matching self-contained notebook · Read the notebook preview

What you will build

Sometimes two responses address the same prompt but one follows the review policy better. Preference data records that comparison. We will calculate one reference-adjusted loss, then train a deliberately tiny policy with two response candidates per prompt.

This policy has no text encoder and cannot generate a new answer. That restriction lets us inspect policy logits, the fixed reference, and the objective directly. It is not a complete generative DPO training system.

By the end: Calculate a DPO pair loss and optimize a restricted response-choice policy.

What you need before starting

You need basic Python: variables, functions, dictionaries, loops, and imports. When this lesson introduces a class or a library-specific term, its role is explained below.

Use Python 3.12. These examples were executed on Windows with Python 3.12.4. Use the shown interpreter consistently when installing packages and running code. This is an original small CPU teaching experiment, not a downloaded general-purpose LLM.

You do not need to run an earlier article to make this part execute. The complete files and data used here are supplied below. The series order is for learning the ideas progressively.

Understand the idea before the code

A preference pair compares two complete responses to the same prompt. “Chosen” means the reviewer preferred one under an explicit rubric; “rejected” means the less-preferred alternative. It is not automatically a proof that every chosen statement is true.

A policy is the model's rule for assigning probabilities to responses. The reference is a fixed starting distribution used for comparison. In our toy experiment each prompt has only two available response candidates. The trained logits choose between those candidates; the model cannot compose a new sentence or generalize to unseen prompts.

DPO, Direct Preference Optimization, uses a loss based on how the policy's chosen-versus-rejected preference differs from the reference's. A log probability is the natural logarithm of a probability. Subtracting two log probabilities gives the log of their ratio. A margin here is the difference between the policy's log-ratio and the reference's log-ratio.

The sigmoid function turns a real number into a value between zero and one: 1/(1+exp(-z)). The loss uses negative log sigmoid of the scaled margin. beta is the scale in that objective; it is a separate configuration value from the optimizer's learning rate.

A picture of this part's idea

This picture introduces the mechanism. The numbers and assertions in the program below are the actual experiment.

Create the lesson folder

Create this folder and keep the filenames exactly as shown. In particular, a local module is an ordinary file beside the program, not an extra package you must guess how to install.

lesson-folder/
  lesson.py
  requirements.txt

Save requirements.txt

These are the external Python packages for this lesson. The local files above are included in the article; they are not pip packages.

torch==2.2.2
numpy==1.26.4
matplotlib==3.9.1

python -m venv .venv
.venv\Scripts\python.exe -m pip install -r requirements.txt

On Linux/macOS, use python3 -m venv .venv and .venv/bin/python -m pip install -r requirements.txt. Installation requires internet access. The default examples do not call a paid API.

Complete code for this article

Save this complete file as lesson.py

This is the complete program, including imports, data, function definitions, and the entry point.

import torch
from torch.nn import functional as F


def seed(value=42):
    torch.manual_seed(value)
    torch.set_num_threads(1)


def dpo_loss(policy_logits, reference_logits, beta=0.1):
    """Two complete candidate responses per prompt, chosen at index zero."""
    logp = policy_logits.log_softmax(-1)
    logr = reference_logits.detach().log_softmax(-1)
    margin = logp[:, 0] - logp[:, 1] - (logr[:, 0] - logr[:, 1])
    return -F.logsigmoid(beta * margin).mean()


def main():
    import torch

    policy = torch.tensor([[0.6, 0.4]]).log()
    reference = torch.tensor([[0.5, 0.5]]).log()
    loss = dpo_loss(policy, reference, beta=0.1)
    print("one-pair DPO loss:", round(float(loss), 3))
    assert abs(float(loss) - 0.673079) < 0.001
    seed()
    pairs = [
        {
            "prompt": "Camera will not start",
            "chosen": "Ask for board and power details",
            "rejected": "Invent an exact wiring fix",
        },
        {
            "prompt": "Linux permission error",
            "chosen": "Ask for command and file permissions",
            "rejected": "Recommend making every file writable",
        },
    ]
    logits = torch.nn.Parameter(torch.zeros(len(pairs), 2))
    reference = torch.zeros_like(logits).detach()
    snapshot = reference.clone()
    optimizer = torch.optim.Adam([logits], lr=0.1)
    losses = []
    for _ in range(100):
        loss = dpo_loss(logits, reference, beta=0.2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(float(loss.detach()))
    print("chosen probabilities:", logits.softmax(-1)[:, 0].detach().tolist())
    assert torch.equal(reference, snapshot)
    assert losses[-1] < losses[0]


if __name__ == "__main__":
    main()

Run it and compare the result

Run commands from the lesson folder. In the commands below, python means the environment you installed into. On Windows, substitute .venv\Scripts\python.exe; on Linux/macOS substitute .venv/bin/python. Do not create the environment with one interpreter and run the lesson with another.

Run 1: python lesson.py

python lesson.py

Captured standard output:

one-pair DPO loss: 0.673
chosen probabilities: [0.9999978542327881, 0.9999978542327881]

The displayed output was captured from the supplied default program in a separate folder, with no project-level module on its import path. It is a reproducible demonstration on these examples, not a claim about arbitrary inputs.

Walk through what just happened

1. Calculate log-ratios

The example uses chosen probability 0.6 versus rejected 0.4, with reference probabilities 0.5 and 0.5. The printed loss is about 0.673.

2. Freeze the reference

detach prevents gradients through the reference values. A saved copy is compared after training so we do not accidentally learn both sides of the objective.

3. Interpret rows and columns

One row belongs to one prompt; column zero is chosen and column one is rejected. log_softmax operates on that final two-candidate dimension.

4. Read the result honestly

A high chosen probability shows that this table learned its supplied preferences. It says nothing about unseen prompts, factual accuracy, or a generative language model’s safety.

Change one thing yourself

Reverse one preference label and inspect the learned probabilities. Explain why stronger optimization cannot correct an incorrect human label.

If your result is different

Beta scales the objective’s reference-adjusted margin. It is separate from learning rate and is not a percentage of politeness.

Check the complete imports and filenames first. Then compare your input and command with the shown run. Numeric results may differ slightly between platforms; the program's assertions check the intended relationships rather than relying on a decorative screenshot.

What you can now explain

Calculate a DPO pair loss and optimize a restricted response-choice policy. Next we will build on this result in Part 9.

Technical reference: PyTorch learning basics. This project and its wording are original CodeTips Academy material.