Base models just predict the next word. Alignment techniques like Reinforcement Learning from Human Feedback (RLHF) teach them to act as assistants, prioritizing helpfulness, honesty, and harmlessness.

Module 1: The RLHF Pipeline

Steps in RLHF

  • 1. Supervised Fine-Tuning (SFT): Train the base model on high-quality instruction-response pairs.
  • 2. Reward Model Training: Humans rank multiple model responses. Train a model to predict these rankings.
  • 3. PPO Optimization: Use Proximal Policy Optimization to adjust the LLM to maximize the Reward Model's score.

RLAIF (AI Feedback) replaces the costly human ranking step with a stronger LLM acting as the judge.