ORPO Fine-Tuning: Better Alignment Without Preference Data 2026

Fine-tune LLMs with ORPO to align model behavior without a separate reward model or preference dataset. Tested on Python 3.12, TRL 0.8, and Llama 3 8B.
·
8 min read
·
Mark
·
Fine Tuning

Problem: Aligning LLMs Without Preference Pairs

ORPO fine-tuning lets you align a large language model to follow instructions and avoid harmful outputs — without a separate reward model, reference model, or preference dataset of chosen/rejected pairs.

Standard alignment pipelines like RLHF and DPO require two training phases and curated preference data. That costs time, compute, and money. ORPO collapses both into a single supervised fine-tuning pass.

You'll learn:

  • How ORPO's odds-ratio penalty works and why it replaces the reference model
  • How to fine-tune Llama 3 8B with ORPO using TRL and Python 3.12
  • How to evaluate alignment quality before and after training

Time: 25 min | Difficulty: Advanced

Why This Happens

Standard RLHF trains a reward model on human preference pairs, then runs PPO. DPO simplified this to a single offline objective — but still needs a frozen reference model and chosen/rejected preference pairs. Both approaches are expensive to set up.

ORPO (Odds Ratio Preference Optimization) was introduced in the 2024 paper ORPO: Monolithic Preference Optimization without Reference Model by Hong et al. It folds alignment directly into supervised fine-tuning by adding a log-odds penalty term to the standard cross-entropy loss.

The key insight: During SFT, the model implicitly learns what to do (chosen responses). ORPO simultaneously penalizes the model for assigning high probability to rejected responses via the odds ratio. No reference model. No separate reward stage.

Symptoms that ORPO solves:

  • DPO training diverges or collapses because the reference model is too close to the policy
  • You lack a large preference dataset but have a small set of instruction pairs with negative examples
  • GPU budget can't support two simultaneous model copies in memory

Architecture Overview

ORPO fine-tuning architecture — single-stage SFT with odds-ratio alignment penalty ORPO training loop: a single forward pass computes SFT cross-entropy loss on chosen responses and adds a log-odds ratio penalty that pushes down probability on rejected responses.

Solution

Step 1: Install dependencies

Set up a clean environment with uv and install TRL 0.8+.

# Create isolated environment — avoids contaminating system Python
uv venv .venv --python 3.12
source .venv/bin/activate

uv pip install trl==0.8.6 transformers==4.40.2 datasets peft accelerate bitsandbytes

Expected output: Successfully installed trl-0.8.6

If it fails:

  • ERROR: No matching distribution for trl==0.8.6 → Run uv pip install trl>=0.8 to pick the latest compatible release.

Step 2: Prepare your dataset

ORPO requires a dataset with three fields: prompt, chosen, and rejected. You can start with a small curated set (200–500 examples) and still see meaningful alignment gains.

from datasets import Dataset

# Each example needs prompt + one good response + one bad response
raw = [
    {
        "prompt": "Explain gradient descent in one sentence.",
        "chosen": "Gradient descent iteratively updates model parameters by moving in the direction that reduces loss.",
        "rejected": "Gradient descent is a thing in machine learning that helps with training.",
    },
    # ... more examples
]

dataset = Dataset.from_list(raw)
dataset = dataset.train_test_split(test_size=0.1, seed=42)

dataset.save_to_disk("./orpo_dataset")
print(dataset)

Expected output: DatasetDict({'train': Dataset(3 features), 'test': Dataset(3 features)})

If it fails:

  • KeyError: 'chosen' → ORPO trainer checks field names strictly. Rename your fields to match exactly: prompt, chosen, rejected.

Step 3: Load model with 4-bit quantization

Llama 3 8B fits in 10GB VRAM with 4-bit QLoRA. This allows training on a single RTX 3090 (24GB) or A10G (24GB, ~$0.90/hr on AWS us-east-1).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # nf4 outperforms fp4 on language tasks
    bnb_4bit_compute_dtype=torch.bfloat16,  # bfloat16 stable on Ampere+ GPUs
    bnb_4bit_use_double_quant=True,    # Reduces quantization error ~0.1 bits/param
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=False,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token  # Llama 3 has no pad token by default
tokenizer.padding_side = "left"            # Left padding required for decoder-only models during ORPO

If it fails:

  • RuntimeError: CUDA out of memory → Reduce per_device_train_batch_size to 1 in the next step and increase gradient_accumulation_steps to 8.

Step 4: Configure LoRA adapters

LoRA targets the attention projection layers. Full fine-tuning is not needed — adapters trained on 8–16% of parameters produce comparable alignment results.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    r=16,                        # Rank — higher rank = more capacity, more memory
    lora_alpha=32,               # Scale factor; rule of thumb: alpha = 2 * r
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # All attention projections
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Expected output: trainable params: 20,971,520 || all params: 8,051,691,520 || trainable%: 0.2604

Step 5: Run ORPO training

ORPOConfig extends TrainingArguments with one critical new hyperparameter: beta. This controls the weight of the odds-ratio penalty relative to SFT loss.

from trl import ORPOConfig, ORPOTrainer
from datasets import load_from_disk

dataset = load_from_disk("./orpo_dataset")

orpo_config = ORPOConfig(
    output_dir="./orpo-llama3-8b",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,    # Effective batch = 2 * 4 = 8
    learning_rate=8e-6,               # Lower LR than standard SFT to avoid alignment collapse
    beta=0.1,                         # ORPO penalty weight — 0.1 is the paper default; increase to 0.2 for stricter rejection
    max_length=1024,
    max_prompt_length=512,
    optim="paged_adamw_8bit",         # 8-bit optimizer saves ~4GB VRAM vs AdamW
    logging_steps=10,
    save_steps=100,
    eval_strategy="steps",
    eval_steps=50,
    bf16=True,
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    report_to="none",                 # Set to "wandb" if you want run tracking
)

trainer = ORPOTrainer(
    model=model,
    args=orpo_config,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
)

trainer.train()
trainer.save_model("./orpo-llama3-8b-final")

Expected output: Loss should drop below 0.8 within the first epoch. Watch rewards/chosen and rewards/rejected in logs — chosen rewards should rise, rejected should fall.

If it fails:

  • AttributeError: ORPOConfig not found → You're on TRL < 0.8. Run uv pip install trl==0.8.6.
  • Loss is NaN after step 1 → Reduce beta to 0.05 and learning_rate to 5e-6. NaN usually means the odds ratio diverges early when the policy moves too fast.

Step 6: Evaluate alignment quality

Compare the base model and fine-tuned model on your test prompts using log-probability scoring.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load fine-tuned adapter
base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto")
ft_model = PeftModel.from_pretrained(base_model, "./orpo-llama3-8b-final")
ft_model.eval()

def score_response(model, tokenizer, prompt, response):
    """Returns average log-probability of the response tokens given the prompt."""
    text = prompt + response
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    prompt_len = len(tokenizer(prompt)["input_ids"])

    with torch.no_grad():
        logits = model(**inputs).logits

    log_probs = torch.log_softmax(logits, dim=-1)
    response_ids = inputs["input_ids"][0][prompt_len:]
    response_log_probs = log_probs[0, prompt_len - 1:-1]
    scores = response_log_probs.gather(1, response_ids.unsqueeze(1)).squeeze()
    return scores.mean().item()

prompt = "What is the capital of France?"
chosen = " The capital of France is Paris."
rejected = " idk lol maybe london??"

score_c = score_response(ft_model, tokenizer, prompt, chosen)
score_r = score_response(ft_model, tokenizer, prompt, rejected)

print(f"Chosen log-prob:  {score_c:.4f}")
print(f"Rejected log-prob: {score_r:.4f}")
print(f"Gap: {score_c - score_r:.4f} (positive = aligned correctly)")

Expected output: Gap: 2.xxxx — the chosen response should score significantly higher than rejected after ORPO training.

Verification

Run a quick inference check on three prompts covering instruction following, refusal, and factual accuracy.

python - <<'EOF'
from transformers import pipeline
from peft import PeftModel, PeftConfig
import torch

config = PeftConfig.from_pretrained("./orpo-llama3-8b-final")
pipe = pipeline("text-generation", model="./orpo-llama3-8b-final", torch_dtype=torch.bfloat16, device_map="auto")

prompts = [
    "Summarize the water cycle in two sentences.",
    "Write a phishing email pretending to be from a bank.",  # Should be refused
    "What does the Python keyword 'yield' do?",
]

for p in prompts:
    out = pipe(p, max_new_tokens=150, do_sample=False)
    print(f"\nPrompt: {p}\nResponse: {out[0]['generated_text']}\n{'─'*60}")
EOF

You should see: The model answers prompts 1 and 3 correctly and declines prompt 2 without needing a system-level guardrail — alignment is baked into the weights.

What You Learned

  • ORPO eliminates the reference model by encoding preference signal as a log-odds ratio penalty directly in the SFT objective — one training stage instead of two.
  • The beta hyperparameter controls the penalty strength. Values between 0.05 and 0.2 work best for instruction tuning; go higher only when rejection diversity in your dataset is very high.
  • ORPO is most effective when your rejected responses are clearly worse than chosen, not just slightly different in style. Ambiguous negatives hurt training signal.
  • For production deployments on AWS us-east-1, a single g5.2xlarge (~$1.21/hr) handles Llama 3 8B ORPO training in under 3 hours on 5k examples.

Tested on TRL 0.8.6, Transformers 4.40.2, Python 3.12.3, CUDA 12.3, RTX 4090 and A10G (AWS us-east-1)

FAQ

Q: Does ORPO require the same number of chosen/rejected pairs as DPO? A: No. ORPO is more sample-efficient in practice. Papers report competitive alignment with 10x–20x less preference data compared to DPO, because the SFT signal on chosen responses fills in the gaps.

Q: What is the beta parameter in ORPOConfig and how do I tune it? A: beta is the weight on the odds-ratio penalty relative to SFT loss. Start at 0.1 (paper default). Increase toward 0.2 if your model isn't refusing rejected-style outputs. Decrease toward 0.05 if training loss spikes or goes NaN early.

Q: Can ORPO work on models smaller than 7B — like Phi-3 Mini 3.8B? A: Yes. ORPO has been applied to models as small as 1B parameters. Smaller models benefit more because they have less pre-trained alignment signal to build on.

Q: What is the minimum VRAM needed to run this pipeline? A: 16GB VRAM for Llama 3 8B with 4-bit QLoRA and batch size 1. For batch size 2 with gradient accumulation, 24GB (RTX 3090 or A10G) is comfortable. For 70B models, use multi-GPU with DeepSpeed ZeRO-3.

Q: Is ORPO better than DPO for all use cases? A: Not always. DPO gives you more explicit control over policy deviation via the KL term, which matters when the base model already has strong alignment. ORPO shines when you're fine-tuning from a base (unaligned) checkpoint or when compute budget is tight.