In This Handbook

How are LLMs Trained?

We’ve treated the LLM somewhat like a black box so far, but to be an informed practitioner or user of generative AI, it helps to understand how these models are created. This lesson digs into the training process of LLMs. We’ll cover the lifecycle: massive pretraining on text, then the post-training steps like instruction tuning and alignment (which we touched on earlier). We’ll clarify what training can do (giving the model capabilities and knowledge) and what it cannot do (guarantee up-to-date info, enforce rules without alignment, etc.). We’ll also distinguish clearly between training vs. prompting vs. system-level controls, so you understand which mechanism addresses which aspect of AI behavior. By the end, you should have a clear mental picture of an LLM’s journey from ingesting data to being deployed in an application, and the roles of the people and processes involved at each stage.

Pretraining: Learning from a massive corpus

The heart of an LLM’s power comes from its pretraining. In this phase, the model is initialized (usually with random weights) and then shown an enormous amount of text, learning to predict tokens. This stage is self-supervised, meaning the training data doesn’t require human labels – the text itself provides context and next-token targets. The model essentially plays “fill in the blank” or “predict the next word” millions or billions of times.

A concrete example: if the training data has the sentence “The cat sat on the [MASK].” (mask representing a missing word), the model will adjust its weights to make “mat” (and other plausible words) more likely in that context. Through zillions of such predictions, it learns grammar, facts, and even some reasoning patterns implicitly. Modern LLMs are typically based on the Transformer architecture, which allows them to attend to (consider) many words of context at once when making each prediction. This architecture scales well with data and compute, which is why as we feed more data and use more computation (and bigger models with more parameters), performance keeps improving.

ⓘ Three ingredients behind LLM capabilities

An LLM’s capabilities emerge from the combination of three distinct factors: the training data (what the model sees), the training objective (predicting the next token), and the model architecture (such as Transformers). None of these alone explains the model’s behavior. It’s their interaction that gives rise to modern LLM performance.

Data for pretraining

Training data is crucial, encompassing a vast selection of the internet: Wikipedia, digitized books, news, forums, and public code, to broadly represent language use. For models like GPT-3, sources included “Common Crawl” (a web snapshot) and curated texts. The data often contains hundreds of billions of tokens. Training is computationally intensive, costing millions of dollars and requiring thousands of GPUs for weeks, which is why only major companies and research labs initially trained the largest models.

During training, the model uses validation data to tune hyperparameters and prevent overfitting. The aim is to minimize prediction error, not to memorize text. Pretraining results in a rich statistical language representation, allowing the model to statistically predict sentence continuation. It also acquires factual knowledge, such as the correlation between “Einstein” and “theory of relativity,” or “Python” and programming-related terms like “code” and “function.”

ⓘ What learning does (and doesn’t) mean

When an LLM “learns,” it does not store facts as entries in a database. Instead, training adjusts billions of numerical weights so that certain patterns become more likely in certain contexts. Knowledge is distributed across the model’s parameters as statistical associations, not stored as explicit facts or rules.

Pretraining has limitations: the model cannot access information or events that occurred after its training data cutoff (e.g., post-2022 events if trained up to 2022) and has no internet access during generation. Furthermore, pretraining alone doesn’t ensure the model follows instructions or avoids inappropriate content; it merely learns to mimic text, including any toxic patterns in the training data. This necessitates the subsequent alignment phase.

ⓘ Distribution shift

LLMs perform best on inputs that resemble the data they were trained on. When prompts or tasks fall outside that distribution, a phenomenon known as distribution shift, performance can degrade sharply. Many surprising failures occur not because because it is operating far from familiar patterns.

But pretraining is the foundation. It gives the model its general capabilities, such as, understanding of many languages, basic facts about the world, the ability to write code, etc., all come from seeing examples during pretraining. For example, GPT-4’s ability to solve coding problems is because GPT-4 saw enough programming and problem descriptions in its training to infer how to write code in context. Pretraining is like the education of the model, where it absorbs as much as possible.

Visualizing the LLM lifecycle, pretraining is like the model’s formative education spanning grade school through college, reading everything. The subsequent fine-tuning and alignment are like finishing schools or job training that tailor the model for specific roles or etiquette.

ⓘ Knowledge vs. behavior

Pretraining primarily gives the model knowledge and general capabilities. Post-training steps mainly shape how that knowledge is expressed. They do not significantly expand the model’s factual knowledge; instead, they influence task-following, tone, safety, and conversational behavior.

Instruction tuning and alignment

After pretraining, we have a model that’s knowledgeable but raw. It doesn’t necessarily do what a user wants without being coaxed, and it might respond in ways we consider unsuitable if prompted naively.

This is where additional training steps come in:

Instruction tuning (Supervised fine-tuning): Developers fine-tune pretrained models by using datasets of prompt-output pairs (e.g., “Translate to French: I love programming” -> “J’adore programmer”). This process, essentially continued training, aims to make the model’s output match the “ideal” output for given instructions. This is what turns a model into an “assistant” capable of interpreting and following user requests. Early efforts, such as Google’s FLAN and OpenAI’s InstructGPT, utilized relatively small datasets (hundreds of thousands of pairs) for this instruction tuning. Consequently, the model is much more likely to respond with the requested task (like summarizing an article) rather than generating unrelated text.

Alignment with human preferences (RLHF): RLHF is a human-in-the-loop feedback process. Humans rank model outputs (e.g., for politeness and factual accuracy), and a reward model is trained to predict these human preferences. The original model is then optimized via reinforcement learning to maximize this reward, teaching it to align with conversational norms and reduce toxic or biased outputs. RLHF significantly improved alignment (as seen in ChatGPT) but can occasionally lead to “over-alignment” (e.g., unnecessary apologies or overly cautious refusals).

ⓘ Evaluating LLMs Is context-dependent

Different stages of training optimize for different goals. Pretraining minimizes prediction loss, alignment optimizes human preference scores, and deployment success depends on task-specific criteria. There is no single metric that captures “model quality” across all uses.

Other post-training refinements: Companies often add post-training safety layers like external filters or heuristics, such as a list of banned content (slurs, extremist material) to refuse disallowed output. They may also apply targeted fine-tuning for specific skills, like making a model more conversational (dialogue transcripts) or domain-specific (medical Q&A), though this overlaps with specialization fine-tuning (covered later).

Alignment, which is statistical, won’t catch every undesirable output or prevent all overly cautious refusals of harmless requests. It’s an ongoing effort to make models helpful, safe, and balanced, essentially giving the model a value function on top of its base knowledge.

Training establishes knowledge/skills, instruction tuning dictates task fidelity, and alignment sets behavioral boundaries/style for practical AI assistants.

What training can and cannot do

Now, an essential perspective: what kinds of problems can be solved by further training (including fine-tuning and alignment), and what problems cannot be solved by just training the model more?

Teach new information (with caveats): Fine-tuning on data about events that occurred after the original training (e.g., COVID-19 articles) can incorporate new knowledge, though it may require significant data and risk trade-offs.
Improve performance on specific tasks: Fine-tuning on domain-specific data (e.g., medical Q&A or coding problems) can significantly boost accuracy and make the model specialized.
Alter the model’s style or tone: Using instruction or conversational data can shift the model’s default behavior, making it more formal, terse, or concise (e.g., how OpenAI fine-tuned GPT-3 to create ChatGPT).
Ingrain constraints or formats: Fine-tuning on specific input/output formats (e.g., only JSON) can train the model to follow those formats by default, eliminating the need for repeated instruction.

However, training cannot do (or is not ideal for) the following:

Ensure factual correctness on open-ended knowledge: Models can get facts wrong or become outdated because training is periodic, not continuous. Retrieval systems (live knowledge bases) are better for up-to-date facts. More data doesn’t guarantee truth; it just absorbs what’s in the data, including biases.
Guarantee adherence to all human values or complex constraints: Alignment is imperfect. It’s impossible to 100% train away all harmful outputs. Training too hard for safety can reduce the model’s usefulness (making it overly cautious). External filters or human review may always be needed.
Work easily with very small data or specific user-specific customization: For small datasets, prompting (few-shot) is often better than fine-tuning (which can overfit). User-specific customization is typically handled via prompts or system settings, not by training a unique model for each user. Training is a global change.
Real-time adaptation: Training is an offline, batch process. The model does not dynamically improve during deployment; changes require developers to gather new data and retrain periodically. Live systems use external user feedback loops for continuous learning.

ⓘ Training vs. runtime feedback

Training is an offline, batch process. Models do not update their weights during normal use, even if users provide corrections. Continuous improvement in deployed systems relies on external feedback loops, data collection, and periodic retraining.

Interpretability/transparency: A model’s vast matrices of weights are a black box, making it hard for humans to interpret where concepts are stored. Increased complexity in larger models often worsens this. When an aligned model refuses a query, its generic explanation is scripted, not a genuine, verifiable articulation of its internal “reasoning.” Users and developers must accept that LLMs cannot explain their true internal logic.

Training vs. prompting vs. system control

It’s key to distinguish between these methods of controlling an AI model:

Training (including fine-tuning): Permanently changes the model’s weights and capabilities. It is global, done by developers using data and compute before use.
Prompting: Temporarily conditions the model for one output or session without changing weights. It is done by the user/application at runtime and is immediate but forgotten quickly.
System-level control: External processes around the model at runtime (e.g., retrieval augmentation, post-processing rules, external critique). It’s everything outside the model’s “brain” used to influence the outcome.

To drive home the difference through a clear scenario, consider you have a base model that sometimes gives too much of a lengthy answer and you want concise answers:

Solution A (Prompting): For each query, add “Keep the answer under 100 words.” This works per query but you have to remember to do it every time (or design your interface to automatically add it).
Solution B (Fine-tuning): Gather a dataset of query -> concise answer pairs, fine-tune the model so it just tends to produce shorter answers by default. Now even without an explicit instruction, it might be concise. But this could sacrifice detail when detail is actually needed, so you have to balance.
Solution C (System level): After the model produces an answer, if it’s longer than 100 words, automatically truncate or summarize it with another model/tool. This doesn’t change the model or the prompt; it’s an external fix.

Each approach has trade-offs. Often, the combined approach is best: e.g., fine-tune to generally not ramble, and also allow a user to override with a prompt if they do want a longer answer (“give me more details”).

LLM lifecycle

To illustrate the LLM lifecycle, imagine a flowchart:

Data Collection: Huge text corpus prepared.
Pretraining: Model trained on this data with self-supervised learning (predict next token). Result: a base model.
Fine-Tuning (Instruction): Further train model on Q&A or instruction-following data. Result: a model that follows prompts better (e.g., “GPT-3” becomes “GPT-3 Instruct”).
Alignment (RLHF): Use human feedback and reinforcement learning to align model with desired behavior. Result: an aligned assistant model (e.g., ChatGPT).
Deployment: Model is deployed behind an interface/API. Now comes prompting – users interact, their prompts combined with system prompts for context/policy. The model generates outputs.
Monitoring & Iteration: Developers watch how it performs. If issues arise (e.g., a certain kind of question it handles poorly or a new popular topic emerges), they might go back and collect more data, then fine-tune or adjust the system prompts/rules, or even do a new round of training for improvement. And the cycle continues.

Understanding this pipeline helps in diagnosing where a problem should be fixed. For instance, if the model often gives wrong info about a particular recent event, you could either (a) update the model via training on that info, or (b) adjust the system to fetch answers from a database for that question, or (c) prompt engineering to encourage the model to say “I’m not sure about recent events.” The right approach depends on the situation (like how frequently that issue comes up, how critical accuracy is, etc.).

In the final lesson, we’ll zero in more on fine-tuning specifically: when it’s worth doing, different techniques for it, and its trade-offs with other methods like retrieval. But the fundamental understanding from this lesson is that training is how models gain their base abilities, and prompting/system design is how we apply those abilities. The line between training time and runtime is important: one is expensive but gives general capability, the other is cheap and gives instance-specific control.

Previous Lesson Next Lesson

Fahim Ul Haq

I’m Fahim, a software engineer specializing in generative AI, system design, and responsible AI adoption.

View Profile

Agentic AI