For a long time I mixed all of this up. Exactly one thing was clear in my head: there's SFT, where you train the model to predict token after token. After that it turned into soup. RLHF, PPO, DPO, RLVR, GRPO, reward model, value function. A pile of three- and four-letter acronyms all orbiting "fine-tuning", and it was never clear which one was a cousin of which, and which was from a different universe entirely. I'm an ML Engineer. My job isn't to "explain how someone else's paper works", it's to design an architecture for a task, train it, ship it to production, and measure whether the business metric actually moved. From that seat, what I was missing was one simple map: what comes after what, what here is supervised and what is real RL, and why a reward model has nothing to do with RL. This article is that map. By the end you'll have what I was missing: what post-training is made of, where the line runs between "we trained a reward model" and "we spun the policy with RL", and how DPO differs from RLHF in plain terms rather than by a symbol in a formula. The big picture: post-training is not a single button First, notation. Let x be the prompt and y the completion (what the model generated in response). Almost everything below is a function of the pair (x, y). How it actually goes: 1. Pretraining. Teach the model language and the general skill of continuing text. It's not an assistant yet, just a giant autocomplete. 2. SFT (a.k.a. IFT, instruction tuning). Teach it not to keep continuing the sentence but to answer in a question-answer format. This is the foundation, nothing downstream works without it. 3. Preference alignment (RLHF). We want the model
How LLMs Are Trained After Pretraining: SFT, Reward Models, and RL Without the Alphabet Soup
Read the original article
hackernoon.com →