Scope
Changing the weights of a model so it behaves the way you want. The decision framework (prompt vs RAG vs fine-tune), supervised fine-tuning (SFT) and why a wrong chat template silently ruins it, parameter-efficient methods (LoRA, QLoRA, DoRA/QDoRA), full fine-tuning, the RLHF pipeline and the direct preference family that replaced most of it (DPO, ORPO, KTO, SimPO), RLAIF / Constitutional AI, GRPO (the DeepSeek-R1 engine), managed Reinforcement Fine-Tuning, distillation, data prep, catastrophic forgetting, and multi-adapter serving. For tuning the model at inference time (no training), see 02 - Decoding, Sampling & Reasoning Controls.
Related: Gen AI Engineer, 07 - RAG, 13 - Evaluation & Evals, 17 - Deployment & Serving
Fine-tuning reshapes how the model responds; RAG injects what it knows at query time — pick by which one your problem actually is.
1. The decision framework: prompt → RAG → fine-tune
Before touching weights, climb the ladder. Each rung is cheaper, faster to iterate, and easier to roll back than the next.
| Technique | Changes | Best for | Cost / iteration |
|---|---|---|---|
| Prompt / context engineering | nothing (inference-time) | most tasks; first thing to try | minutes |
| RAG | nothing — supplies external knowledge at query time | facts that change, are private, or are too big for the prompt | hours |
| Fine-tuning | the weights — bakes in behaviour, style, format, skill | consistent tone/format, narrow task mastery, latency/cost reduction, distillation | days + GPUs |
The one rule to never get wrong
Fine-tuning changes behaviour; RAG supplies knowledge. If the goal is “the model should know our 4,000 support articles,” that is a RAG problem — fine-tuning to memorize facts is unreliable, expensive, and goes stale the moment a fact changes. Fine-tune when you want the model to act a certain way (always emit this JSON shape, adopt this voice, reason in this domain’s style, call tools this way). The two compose: fine-tune for behaviour, then RAG for fresh facts.
Cheap signal in interviews
“We’ll fine-tune so it knows our docs” is the single most common wrong answer. The right reflex: what kind of gap is this — knowledge (RAG) or behaviour (fine-tune)? Reach for fine-tuning only after prompting and RAG have been tried and measurably fall short.
2. Supervised Fine-Tuning (SFT)
The workhorse. Show the model (prompt, ideal_completion) pairs and train it to reproduce the completion via standard next-token cross-entropy.
How it works. Same loss as pretraining, but on curated demonstrations. Two non-obvious correctness requirements decide whether it works at all:
- Loss masking. You only compute loss on the completion (assistant) tokens, not the prompt tokens. The prompt is context to condition on, not something to learn to generate. Train on both and you waste capacity teaching the model to parrot user inputs and degrade instruction-following.
- Chat-template correctness. Each base/instruct model expects an exact wrapper around turns — special tokens like
<|im_start|>,[INST], role markers, BOS/EOS placement. You must format training data with the same template the model was aligned with (and the same one inference will use).
A wrong chat template is a silent failure
Mismatched special tokens or role markers don’t crash — training loss drops, everything looks fine, and the model just learns slightly wrong behaviour that surfaces as degraded quality you can’t easily trace. Always use the tokenizer’s built-in
apply_chat_template, verify a few rendered examples by eye, and confirm EOS is present so the model learns to stop. This is the most common quiet bug in SFT.
| Pros | Cons |
|---|---|
| Direct, well-understood, stable | Needs high-quality labelled demonstrations |
| Strong for format/style/tool-call consistency | Only as good as the data; garbage in, garbage out |
| Foundation for everything downstream (preference tuning starts from an SFT checkpoint) | Can overfit a narrow distribution → catastrophic forgetting (§11) |
3. PEFT: LoRA, QLoRA, DoRA / QDoRA
Full fine-tuning updates all weights — billions of parameters, huge optimizer state, many GPUs. Parameter-Efficient Fine-Tuning freezes the base model and trains a tiny number of new parameters instead.
LoRA — Low-Rank Adaptation
Freeze the base weight W; learn a low-rank update ΔW = B·A where A and B are skinny matrices of rank r. Train ~0.1–1% of parameters; at inference W + ΔW (can be merged for zero added latency). Adapters are tiny (MBs) and swappable.
QLoRA — Quantized LoRA
Load the frozen base in 4-bit NF4 (NormalFloat, tuned for the normal distribution of trained weights), add double quantization (quantize the quantization constants, saves ~0.37 bits/param) and paged optimizers to survive memory spikes. LoRA adapters train in higher precision on top. Result: fine-tune a 65B model on a single GPU with near-full-precision quality. This is what made fine-tuning accessible to non-labs.
DoRA / QDoRA (2024–2025, the emerging default)
DoRA (Weight-Decomposed Low-Rank Adaptation, ICML 2024 Oral) decomposes each weight into magnitude and direction, then applies LoRA only to the directional component. It tracks full fine-tuning’s learning behaviour more closely than LoRA, narrows the LoRA–full gap, and adds no inference overhead (merges back). Flip it on in HF PEFT with use_dora=True. QDoRA = DoRA on a 4-bit base (the QLoRA recipe); with FSDP it beats QLoRA and edges out full fine-tuning on consumer GPUs.
| Method | Trainable | Base precision | Notes |
|---|---|---|---|
| Full FT | 100% | 16-bit | most capacity, most cost |
| LoRA | ~0.1–1% | 16-bit | the baseline PEFT; mergeable |
| QLoRA | ~0.1–1% | 4-bit NF4 | single-GPU large models |
| DoRA | ~LoRA | 16-bit | magnitude/direction split; closer to full FT |
| QDoRA | ~LoRA | 4-bit | DoRA + quantized base; QLoRA killer |
Hyperparameters that actually move the needle
- rank
r— capacity of the adapter. 8–16 for style/format; 32–64+ for harder skill transfer. Bigger isn’t always better — it can overfit.- alpha
α— scaling; the effective update scalesα/r. Common heuristicα = 2r(QLoRA paper usedr=64, α=16). Tune one, hold the other.- learning rate — LoRA tolerates higher LRs than full FT (e.g. 1e-4–3e-4) since far fewer params move.
- target modules — apply to all linear layers (incl. MLP), not just attention
q,v; broader coverage closes more of the quality gap.
4. Full fine-tuning — when it’s actually justified
Updating all weights is rarely the right first move, but it’s correct when:
- You’re doing continued pretraining / large domain adaptation (a new language, code dialect, or domain with its own token distribution).
- You have abundant high-quality data and the budget; you need the last few points of quality PEFT can’t reach.
- You’re distilling into a small model you’ll own and serve at scale (§10).
| Pros | Cons |
|---|---|
| Maximum capacity / ceiling quality | Expensive: many GPUs, full optimizer state |
| No adapter routing at serve time | Highest catastrophic-forgetting risk |
| One checkpoint per task — no cheap multi-tenant adapters |
PEFT first
For most product fine-tuning (tone, format, narrow task, tool use), LoRA/QLoRA/DoRA reaches ~full-FT quality at a fraction of the cost and gives you swappable adapters. Justify full FT explicitly — don’t default to it.
5. RLHF — the classic 3-stage pipeline
How preferences (not just demonstrations) get into a model. The original ChatGPT/InstructGPT recipe:
- SFT — fine-tune on demonstrations (§2) to get a reasonable instruction-follower.
- Reward Model (RM) — collect human rankings of pairs of outputs; train a model to predict which a human prefers (a scalar reward).
- RL optimization (PPO) — optimize the SFT policy against the RM with PPO, plus a KL penalty to the SFT reference so it doesn’t drift into reward-hacked gibberish.
| Pros | Cons |
|---|---|
| Captures fuzzy “what humans prefer” no label can express | 3 models in memory (policy, ref, RM ± critic); complex, unstable |
| Behind the first wave of aligned assistants | Expensive human preference data; reward hacking |
PPO-only RLHF is now the legacy answer
Saying “RLHF = PPO” in 2026 signals stale knowledge. PPO still works and labs still use it, but for most teams it’s been displaced by direct preference methods (§6) and, for reasoning, GRPO (§8). Know PPO as the foundation; don’t present it as the default.
6. Direct preference methods — DPO, ORPO, KTO, SimPO
The big simplification: skip the separate reward model and PPO loop — optimize the policy directly on preference data with a classification-style loss. Far simpler, more stable, cheaper.
| Method | Idea | Needs | When |
|---|---|---|---|
| DPO | Reparameterize the RLHF objective so the policy is its own implicit reward; train directly on (chosen, rejected) pairs against a frozen reference. β controls deviation from reference. | SFT checkpoint + ref model + paired prefs | the default; clean, well-supported |
| ORPO | Fold preference into SFT — one stage, no reference model, an odds-ratio penalty on the rejected response added to the SFT loss. | paired prefs (SFT + pref in one pass) | skip the separate SFT stage; lean pipelines |
| KTO | Kahneman-Tversky: learn from unpaired thumbs-up / thumbs-down signals — no need to match chosen with rejected. | binary good/bad labels (no pairs) | you have production 👍/👎 logs, not curated pairs |
| SimPO | Reference-free; length-normalized average log-prob reward + a target margin. Aligns the training reward with the generation objective and avoids length bias. | paired prefs, no ref model | memory-lean, fights verbosity |
How to choose
Start DPO if you have clean pairs. ORPO to collapse SFT+pref into one cheaper stage. KTO when your data is naturally binary (real product feedback) rather than paired. SimPO when reference-model memory hurts or outputs run long.
β(DPO/SimPO) is the key dial: higher = stay closer to the reference (safer, less movement); lower = move more (riskier, can reward-hack).
7. RLAIF & Constitutional AI
Human preference labels are slow and expensive. RLAIF (RL from AI Feedback) replaces the human labeler with an LLM judge that ranks outputs against a rubric — scaling preference data cheaply. Constitutional AI (Anthropic) goes further: the model critiques and revises its own responses against a written set of principles (a “constitution”), generating its own preference data, so alignment scales with compute instead of headcount. The trade-off: the judge model’s biases and blind spots propagate into the trained model — RLAIF is only as good as the rubric and the grader.
8. GRPO — the DeepSeek-R1 reasoning engine
The 2025 breakthrough for reasoning
GRPO (Group Relative Policy Optimization) is the RL algorithm behind DeepSeek-R1. Its key move: drop the critic/value network PPO needs. Sample a group of
kanswers for one prompt, score them, and use the group’s mean as the baseline — each answer’s advantage is just how much better/worse than its peers. Halves the model count vs PPO and cuts memory.
What makes it powerful is the reward: verifiable / rule-based rewards. For math, code, and structured tasks the grader is a regex / unit test / string match (R1-Zero used no neural reward model at all) — the reward is the correct answer, not a learned preference. No reward model to hack, and the model discovers reasoning chains on its own.
| Pros | Cons |
|---|---|
| No critic → simpler, cheaper than PPO | Needs an automatically verifiable signal (math/code/format) |
| Verifiable rewards can’t be neural-hacked | Less natural for fuzzy/subjective quality |
| Produced open frontier-level reasoning (R1) | Still RL — needs KL control and tuning |
9. Managed Reinforcement Fine-Tuning (RFT)
GRPO-style RL, productized — you bring a programmable grader, the platform runs the RL loop.
- OpenAI RFT — fine-tune reasoning models (o4-mini) with a feedback signal you define instead of fixed answers. Supports code-based and model-based graders and structured outputs; available to verified orgs. Reported wins: SafetyKit F1 86%→90%, others +12–25 points.
- Amazon Bedrock RFT — RFT via OpenAI-compatible APIs; orchestrates verifiable-reward or LLM-as-judge grading across thousands of prompt-response pairs, handling concurrency/retries.
When RFT beats SFT
Use RFT when you can score a good answer but can’t easily write one — the task has many valid solutions but a checkable success criterion (passes tests, matches a spec, hits a metric). If you can write ideal completions cheaply, plain SFT is simpler. Ties to 17 - Deployment & Serving and 13 - Evaluation & Evals — the grader is your eval.
10. Distillation
Train a small student to imitate a large teacher. The 2025 insight that matters:
Transfer reasoning traces, not just answers
Naive distillation copies final answers. Modern reasoning distillation has the teacher emit full chain-of-thought traces, and the student trains on the reasoning, not only the conclusion. DeepSeek distilled R1’s traces into small dense models (Qwen/Llama) that punch far above their size on reasoning benchmarks. Distillation is often the cheapest path to a fast, ownable model that approximates a frontier one.
| Pros | Cons |
|---|---|
| Small, cheap, low-latency student | Capped by the teacher; inherits its biases/errors |
| Reasoning traces transfer real skill | Needs teacher output generation at scale (cost) |
| You own and serve it | License terms on teacher outputs may restrict use |
11. Data prep, catastrophic forgetting & eval
Data prep is most of the work. Quality and diversity beat volume; dedupe; balance the distribution; enforce the exact chat template (§2); split a held-out set before training; scrub PII/leakage. A few thousand clean, diverse examples usually beat tens of thousands of noisy ones.
Catastrophic forgetting. Fine-tuning hard on a narrow task can erase general capability — the model gets great at your task and worse at everything else. Mitigate by: PEFT (frozen base forgets less than full FT), mixing in a slice of general/instruction data (“replay”), lower LR / fewer epochs, and evaluating general benchmarks alongside your task metric.
Watch the tail, not just the average
A stable or rising average eval score can hide a model that is quietly collapsing on the tail of rare/hard queries — an early-collapse signal. Always slice evals (by query type, difficulty, length, language) and track tail percentiles, not just the mean. A flat average with a sinking p95 means you’re overfitting and about to regress in production. See 13 - Evaluation & Evals.
12. Multi-LoRA serving
One base, many hot-swappable adapters
Because LoRA adapters are tiny and sit on top of a frozen base, you serve one base model in VRAM and load dozens of adapters, selecting one per request. vLLM has supported this since v0.3.0 (
--max-lorascaps how many stay resident; dynamic load/evict via API in v0.6.2+). This is what makes per-customer / per-task fine-tunes economical — fine-tune once per tenant, serve them all on a single GPU instead of one full model each.
Trade-off: a tiny per-request overhead to apply the adapter (vs a merged adapter which has none but can’t be swapped). Choose merged for a single dominant task at max throughput; multi-LoRA for many tenants/tasks on shared hardware. Architecture and routing details in 17 - Deployment & Serving.
13. Pros / cons / when — pick the method
| Situation | Reach for |
|---|---|
| Knowledge that changes or is private | RAG (not fine-tuning) — 07 - RAG |
| Consistent tone / format / structured output | SFT (+ LoRA) |
| Large model, single/few GPUs | QLoRA / QDoRA |
| Want closer-to-full-FT quality cheaply | DoRA |
| Align to human preference, clean pairs | DPO |
| Collapse SFT + preference into one stage | ORPO |
| Only have 👍/👎 product logs | KTO |
| Reference-model memory hurts / fight verbosity | SimPO |
| Reasoning, verifiable answers (math/code) | GRPO / managed RFT |
| Cheap labels at scale | RLAIF / Constitutional AI |
| Fast small model approximating a big one | Distillation (transfer traces) |
| Many tenants on shared hardware | Multi-LoRA serving |
14. Caveats & risk mitigation
- “Fine-tune to ADD knowledge” is wrong. It memorizes unreliably, costs a retrain per fact change, and increases hallucination on edges. Use RAG for knowledge; fine-tune for behaviour. The two compose.
- PPO-only RLHF is stale. Direct preference methods (DPO/ORPO/KTO/SimPO) and GRPO are the modern defaults. Cite PPO as foundation, not as the recommendation.
- Reward hacking. Any learned reward can be gamed (the policy finds the cheat, not the goal). Mitigate with verifiable graders (rules/tests, not neural reward models where possible) and a KL constraint to the reference so the policy can’t drift far for cheap reward. Cap with held-out evals.
- Chat-template / loss-mask bugs are silent. No crash, just quietly worse behaviour. Use
apply_chat_template, mask the prompt, verify EOS, eyeball rendered examples. (§2) - Stable average, dropping tail = early collapse. Slice evals; track tail percentiles; a sinking p95 under a flat mean is overfitting. (§11)
- Catastrophic forgetting. Prefer PEFT, mix replay data, fewer epochs, and benchmark general capability — not only your task metric.
- Cost & ROI. Fine-tuning adds GPU spend, an MLOps surface (versioning adapters, eval gates, rollback), and ongoing retrains. Quantify against a prompt + RAG baseline before committing — often the cheaper rungs already clear the bar.
15. In practice
# QLoRA / DoRA SFT — frozen 4-bit base, tiny trainable adapter
from peft import LoraConfig
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(model_id)
# CRITICAL: use the model's own template; mask prompt tokens; keep EOS
text = tok.apply_chat_template(example["messages"], tokenize=False)
peft_cfg = LoraConfig(
r=16, lora_alpha=32, # effective scale alpha/r; tune ONE
target_modules="all-linear", # not just q,v — closes the gap
lora_dropout=0.05,
use_dora=True, # DoRA: magnitude+direction split
task_type="CAUSAL_LM",
)
# train with completion-only loss (prompt tokens masked), low epochs
# DPO — direct preference, no reward model / no PPO
# loss over (chosen, rejected) pairs vs frozen reference; beta controls drift
# trainer = DPOTrainer(model, ref_model, beta=0.1, train_dataset=pref_pairs)16. Sources
- LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023)
- DoRA: Weight-Decomposed Low-Rank Adaptation (ICML 2024 Oral)
- NVIDIA — Introducing DoRA, a high-performing alternative to LoRA
- Direct Preference Optimization (DPO)
- ORPO: Monolithic Preference Optimization without Reference Model
- KTO: Model Alignment as Prospect Theoretic Optimization
- SimPO: Simple Preference Optimization with a Reference-Free Reward
- DeepSeek-R1: Incentivizing Reasoning Capability via RL (GRPO)
- DeepSeekMath — GRPO introduced
- Constitutional AI: Harmlessness from AI Feedback
- OpenAI — Reinforcement fine-tuning guide
- AWS — Reinforcement fine-tuning on Amazon Bedrock
- vLLM — Using LoRA adapters (multi-LoRA serving)