How Large Language Models Actually Learn: Pretraining, Fine-Tuning, and RLHF

Every time you type a prompt into ChatGPT, Claude, or Gemini, you're talking to a system that was built in three distinct stages. Understanding those stages does more than satisfy curiosity. It explains why the models behave the way they do, and why certain prompts work better than others. Here's how a large language model actually goes from a pile of text to a helpful assistant.

It all starts with the transformer

Modern LLMs are built on the transformer, an architecture introduced in the 2017 paper "Attention Is All You Need" by researchers at Google. Its key idea is self-attention: a mechanism that lets the model weigh how much every word (really, every token) in a sequence should influence every other word. Where the recurrent networks that came before it read text step by step, the transformer processes an entire sequence in parallel, which is what made training on internet-scale data practical. Almost every major model since, including GPT, Claude, Gemini, and Llama, is a descendant of this one design.

Pretraining: learning to predict the next token

The first and most expensive stage is pretraining. The model is fed enormous amounts of text and given a deceptively simple task: predict the next token. A token is a chunk of text, often a word or a fragment of one, produced by a tokenizer using a scheme like byte-pair encoding. By guessing the next token billions of times and adjusting its internal weights whenever it's wrong, the model gradually absorbs grammar, facts, reasoning patterns, and style. Crucially, this is self-supervised: no human has to label the data, because the "answer" is just the next word already present in the text.

The scaling laws that drove the boom

Why did models suddenly get so capable? A big reason is scaling laws. Research from OpenAI in 2020 showed that model performance improves predictably as you increase parameters, data, and compute together. In 2022, DeepMind's Chinchilla paper refined this, arguing that many models were oversized and undertrained, and that for a given compute budget you should train a smaller model on far more data. This shifted the field's focus from raw parameter count toward the quality and quantity of training tokens, and it's part of why newer, smaller models often outperform older, larger ones.

Fine-tuning: from autocomplete to assistant

A freshly pretrained model is essentially a very sophisticated autocomplete. It will happily continue your text, but it doesn't reliably follow instructions or answer questions in a useful format. The fix is supervised fine-tuning (SFT), also called instruction tuning: the model is trained further on curated examples of instructions paired with high-quality responses. This teaches it the shape of an assistant, so it treats your input as a request to fulfill rather than a passage to extend.

RLHF: teaching the model what "good" looks like

The final polish for most chat models is reinforcement learning from human feedback (RLHF), the technique popularized by OpenAI's 2022 InstructGPT work. Humans rank multiple model responses to the same prompt; those rankings train a separate reward model that learns to score outputs the way people would. The language model is then optimized against that reward signal, nudging it toward answers that are more helpful, honest, and harmless. Newer variants like Direct Preference Optimization (DPO) reach similar results with a simpler training setup, but the core idea is the same: use human preferences to steer behavior that raw text prediction can't.

Why context windows keep growing

One number you'll see quoted constantly is the context window, the amount of text a model can consider at once, measured in tokens. It matters because everything the model "knows" in a conversation has to fit inside it. Early models handled only a few thousand tokens. Today's leading models handle hundreds of thousands, and some exceed a million, enough to load entire books or codebases. The challenge is that standard attention scales quadratically with sequence length, so longer contexts demand real engineering to stay fast and affordable. That is why retrieval and smart prompting still matter even as windows expand.

Understanding this pipeline pays off at the keyboard. Knowing that a model is a next-token predictor tuned on human preferences is exactly why clear roles, concrete examples, and explicit formatting make such a difference. Put it into practice with the community-rated prompts on PromptingIndex.