The Finishing Touches: Polishing the AGI Diamond

A pre-trained model (Base Model) is like a wildly intelligent alien that has read the entire internet but has no desire to help you. Ask it 'How to kill a process?' and it might complete the sentence with '...and hide the body.' It predicts the next token. It doesn't care about your feelings or safety.

Post-training is the process of lobotomizing this alien just enough to make it polite, while keeping it smart. It's a delicate surgery.

• SFT (Supervised Fine-Tuning): The 'Monkey See, Monkey Do' phase. Humans write good questions and answers. The model mimics them. This teaches the format.
• RLHF (Reinforcement Learning from Human Feedback): The 'Good Dog, Bad Dog' phase. The model generates two answers. A human picks the better one. A Reward Model learns this preference and trains the main model via PPO (Proximal Policy Optimization).
• RLAIF (RL from AI Feedback): The 'Inception' phase. AI models grade other AI models. This is how we scale, because humans are too slow and expensive.

Here is the spicy part: Alignment makes models dumber. It's called the 'Alignment Tax'. When you force a model to be 'safe' and 'unbiased', you are effectively blocking off neural pathways that might contain creative or edge-case solutions. A perfectly aligned model is a brick—safe, but useless.

DeepSeek R1 introduced GRPO (Group Relative Policy Optimization). Instead of a separate heavy Critic model (standard PPO), they use group scores from multiple outputs. It's more efficient and stable. This is why R1 feels 'rawer' and sometimes smarter than GPT-4—it might have paid less of an alignment tax.