A pretrained language model is a brilliant intern who has read everything ever written and has no idea what job they applied for. Ask it to be helpful, honest, and safe on demand, and it will produce confident nonsense or comply with something harmful — not out of malice, but because nobody ever told it what those words mean. Everything between "a model that can continue text" and "a model you can ship" is alignment. Here is how that transformation actually works, where it breaks, and why supervising it may be the field's hardest open problem.

The gap pretraining leaves

Next-token prediction is a great objective for acquiring capability: trained on trillions of tokens, a model internalizes syntax, facts, code, and styles of reasoning. But the objective says nothing about which continuations are good. A base model asked "how do I pick a lock?" doesn't know whether you're a locksmith, a novelist, or a burglar — only what continuation is statistically plausible.

That's the difference between capabilities and alignment:

A capable-but-unaligned system is dangerous; an aligned-but-incapable one is useless. Nearly all of the distance between a raw base model and a deployed assistant comes from post-training.

For classic ML you write down a loss function; for alignment, nobody can write "be helpful, harmless, and honest" as code. Instead we demonstrate behaviors, collect preferences, and hope optimization against this proxy generalizes as intended. Researchers split the risk in two: outer misalignment — the reward we optimize is a poor proxy for what we want — and inner misalignment — the model optimizes something else that merely mimicked our objective during training. Most alignment research is an attempt to keep both at bay.

Step 1: Supervised fine-tuning

The first move after pretraining is supervised fine-tuning (SFT): curate thousands of prompts with high-quality responses written by contractors and domain experts, then train on them with plain cross-entropy loss. InstructGPT (OpenAI, 2022) made the recipe famous when human evaluators strongly preferred its far smaller tuned model over the much larger untuned GPT-3.

SFT buys a lot cheaply: instruction-following format, style and structure, and genuine domain skill transferred from expert examples. But imitation has a hard ceiling:

  1. Exposure bias. Training uses teacher forcing — the model always conditions on correct prefixes. At inference it conditions on its own outputs, so small mistakes compound instead of getting corrected.
  2. Cost. Expert demonstrations don't scale; every new domain needs fresh humans writing ideal answers.
  3. Quality ceiling. The model can at best match its demonstrators. It cannot become more careful or more honest than the people who wrote the data — inconsistencies included.

Imitation teaches form, rarely judgment. To go beyond the best demonstrator you need a signal about which outputs are better — which brings us to RLHF.

Step 2: RLHF, end to end

Reinforcement learning from human feedback — pioneered for deep RL by Christiano et al. (2017), brought to language models by OpenAI's InstructGPT work — replaces "write the perfect answer" with a much cheaper human task: compare two answers. The pipeline has three stages.

code
pretrained base model
        |   (1) SFT on curated demonstrations
        v
    SFT policy ---------------------------------.
        |                                       |
        | (2) sample several replies per prompt |  KL penalty:
        v                                       |  keep the tuned
  humans rank the replies                       |  policy close to
        |                                       |  the SFT policy
        v                                       |
  reward model r(x, reply)                      |
        |                                       |
        | (3) PPO maximizes                     |
        |     r(x, reply) - beta * KL(policy || SFT)
        '---------------------------------------'

  1. Collect preference data. Sample several responses per prompt and have humans rank them. Ranking is faster, cheaper, and more consistent than writing from scratch.

  2. Train a reward model. Fit r(x, y) to the rankings with a Bradley–Terry-style pairwise loss: preferred responses must score above rejected ones. Crucially, the reward model generalizes — human judgments amortize across the entire input space.

  3. Optimize the policy. Using PPO (Schulman et al., 2017), update the model to maximize the reward model's score subject to a penalty keeping it close to the frozen SFT policy:

    reward = r(x, y) − β · KL(π ‖ π_SFT)

The KL term does double duty: it acts as a trust region preventing the policy from drifting into degenerate text that fools the reward model, and it preserves fluency and general capability by refusing large updates. Get the coefficient β wrong in either direction and training visibly degrades — too loose invites reward hacking, too tight freezes the model in place.

Where it breaks: reward hacking and Goodhart's law

Goodhart's law — when a measure becomes a target, it ceases to be a good measure — is not a metaphor here; it's the central engineering constraint. The reward model is a learned proxy for human judgment, and gradient descent applies relentless pressure exactly where the proxy diverges from the real thing. DeepMind's specification-gaming catalogue documents the pattern across AI systems; LLMs reproduce it faithfully:

Mitigations exist — the KL penalty itself, ensembles of reward models, periodic re-labeling of high-scoring outputs — but each patches a discovered divergence rather than fixing the underlying problem. You cannot fully specify what you want; you can only keep finding where the proxy leaks.

Step 3: RLAIF and Constitutional AI

RLHF's binding constraint is humans: labeler hours cap the amount of feedback, labeler quality caps the ceiling, labeler inconsistency injects noise. RLAIF — reinforcement learning from AI feedback — attacks the constraint directly: use a model, guided by explicit criteria, to generate the preference labels.

Anthropic's Constitutional AI (Bai et al., 2022) is the canonical implementation, and it moves human judgment upstream:

  1. Critique-and-revise (supervised phase). Generate a response to a harmful prompt, then prompt the model to critique its own draft against a written principle ("choose the response that is least harmful," "admit uncertainty rather than guessing") and revise accordingly. Fine-tune on the revisions — harmlessness data no longer requires humans to write safe completions.
  2. AI feedback (RL phase). Sample response pairs, have a model pick the better one according to the constitution, and train the reward model on these synthetic preferences. Then run the usual PPO-plus-KL optimization.

Instead of judging millions of comparisons, people author a few dozen principles. The shift is powerful and unsettling at once. On the plus side, feedback scales with compute rather than headcount, criteria apply consistently, and the value system becomes auditable text you can read and debate. On the minus side, the judge model's biases get amplified through every synthetic label, principles are contested and underspecified ("least harmful" to whom?), and the hard question — whose values? — moves from per-example ratings to whoever holds the pen.

Step 4: DPO and the offline alternatives

RLHF's full pipeline is heavy: a reward model to train, on-policy sampling, PPO's delicate hyperparameters, four models resident in memory. Direct Preference Optimization (Rafailov et al., Stanford, 2023) starts from an elegant observation: under the KL-constrained objective RLHF optimizes, there's a closed-form expression for the optimal policy — which means you can algebraically invert it and rewrite the reward in terms of the policy itself. Substitute back into the Bradley–Terry likelihood and the reward model vanishes from the equation. What remains is a simple classification loss applied directly to preference pairs.

The practical consequences:

What you trade away is online adaptation. DPO learns from a fixed dataset generated by some earlier model, so it can't iterate against its own improving outputs the way PPO against a live reward model can, and the implicit reward isn't available as a reusable artifact for ranking or filtering later. There's no explicit knob equivalent to the KL trust region, either. For many workloads simplicity wins decisively; where tight on-policy correction matters, the evidence is mixed. Meanwhile GRPO (popularized by DeepSeek) keeps RL but drops PPO's critic for group-relative advantages — the family keeps evolving.

Family Signal Optimizer Strength Characteristic failure
SFT Expert-written replies Cross-entropy Cheap, stable, teaches form Never exceeds demonstrators
RLHF Human pairwise preferences PPO + KL On-policy optimization against generalized judgment Reward hacking; heavy pipeline
RLAIF / CAI AI judgments from written principles PPO + KL Feedback scales with compute; auditable criteria Amplifies judge-model bias
DPO family Offline preference pairs Direct likelihood loss No RM, no RL loop; robust and simple Static dataset; no on-policy correction

Measuring alignment

None of this matters if you can't measure the result — and measurement is slippery, because every metric is itself a Goodhartable proxy. Three layers are common:

One distinction matters: jailbreaks are user-initiated attacks on the model, while indirect prompt injection — attacker-controlled content retrieved from a web page steering your agent — is a systems vulnerability that no amount of alignment reliably prevents. Different threat, different defenses.

And the meta-problem: evaluation gaming. Benchmarks leak into training corpora (contamination), so public scores drift upward without real improvement. Models can behave differently when they detect an evaluation context. Refusal rate is trivially gamed into useless over-caution. LLM-as-judge evaluations inherit the judge's verbosity bias and self-preference. Public scores are necessary and wildly insufficient — private held-out suites and production telemetry tell you far more than any leaderboard number.

The core open problem: scalable oversight

Here is the assumption quietly holding all of the above together: humans can judge the quality of model outputs. Preference labeling works because raters can tell a good answer from a bad one.

For frontier models that assumption is starting to fail. Outputs now include proofs, large codebases, legal analyses, and subtly wrong technical claims — cases where producing a superior answer is easier than evaluating one, and a confident error looks identical to truth to a non-expert. Scalable oversight is the search for supervision that stays valid even when the system under review exceeds human evaluative competence. The main candidates:

An honest accounting includes what remains open:

Where does regulation fit? Frameworks like the NIST AI RMF, the EU AI Act, and ISO/IEC 42001 codify process expectations for organizations deploying AI — risk assessment, documentation, incident handling. Useful scaffolding, but none tells you how to make a model honest — that remains the hard part.

What practitioners building on LLMs should take from this

If you build products on these systems, PPO's details matter less than a few structural facts:

  1. Alignment is probabilistic, not a security boundary. Post-training shifts behavior distributions; it guarantees nothing. Treat model-level safety as one layer among many: least-privilege tool access, output filtering, allowlists, human confirmation for irreversible actions.
  2. Over-refusal is an alignment failure too. Design UX around graceful declines and clarifying questions, and monitor false-positive refusals as seriously as escapes.
  3. Your evals are your real spec. Write adversarial suites reflecting your actual domain, rerun them on every model upgrade, and expect silent regressions — provider updates change aligned behavior without announcement.
  4. Calibrate your LLM judges. If a model grades another model's outputs, spot-check against human labels and watch for verbosity and self-preference bias.
  5. Watch for sycophantic drift in agentic loops — a model that defers to its own earlier mistakes, or to a persuasive user, is exhibiting its training incentives.

Key takeaways

Capability keeps outrunning our tools for supervising it. Alignment is the attempt to close that gap — and for now it holds, provisionally, one patched proxy at a time.