How an LLM works, explained without math

Before transformers

Language models are not new. Before 2017, the common neural networks for text were recurrent neural networks (RNNs) and their improved versions, LSTMs (long short-term memory) and GRUs (gated recurrent units).

They read text one word at a time, in order, and carried a kind of running memory from one word to the next. That worked for short text, like one sentence or a short paragraph, which is why early neural machine translation systems used them. Translation is a good example of why order and context matter: "I am the best" has four words in English, but its Spanish translation, "Soy el mejor", has three, so you can't translate word by word, you need the meaning of the whole sentence.

But reading one word at a time had two problems. It was slow, because each step had to wait for the previous one, so it couldn't use the parallel processing of GPUs well. And it was forgetful, because by the end of a long text, the memory of the beginning had faded.

Transformer architecture

In 2017, a team of researchers at Google published a paper called Attention Is All You Need, which introduced the transformer. The transformer is a neural network architecture, and its key idea is self-attention: instead of reading word by word, it looks at all the words in the input at once and, for each word, works out which other words matter most for understanding it.

Because it processes the whole sequence in parallel, it trains much faster on GPUs, and because every word can look directly at every other word, it handles long-range relationships much better than RNNs. Almost every LLM you use today, GPT, Claude, Gemini, Llama, Qwen, DeepSeek, is built on this architecture.

The first wave of transformer language models came in 2018: GPT from OpenAI and BERT from Google. GPT was trained to generate text, predicting the next word, while BERT was trained to understand text by reading it in both directions, which made it very good for tasks like classification and search. GPT-3 in 2020 showed how far scaling this up could go, and ChatGPT, launched in November 2022 on a model from the GPT-3.5 series, made it something everyone could use.

What GPT stands for

The three words in GPT explain most of what an LLM is.

Generative

Generative comes from statistics. Statistical modelling is about approximating things with probability, like the chance it will rain tomorrow, and generative modelling is the branch that generates new data, predicting the next value based on the previous values and their probabilities.

For an LLM, the "next value" is the next token. Machines only understand numbers, and mathematics is the only tool we have to give them abilities that look like ours, so every word a model writes is a number it predicted and then converted back into text.

Pre-trained

A pre-trained model has already been trained on a fundamental task, and you build on top of that instead of starting from zero.

Think about how people learn. You learn how to move each chess piece before you play in a competition. You learn to walk before you run, and to ride a bicycle before you drive. Once you know the fundamentals, learning the specific skill is much faster.

An LLM's fundamental task is understanding language: how words relate, how sentences are formed, and what usually follows what. Once a model has learned that from a huge amount of text, it can be adapted to specific tasks, like answering questions, writing code or classifying documents, with much less data and compute.

This is why almost nobody trains an LLM from scratch. You use the knowledge already stored in a pre-trained model, the same way you learn faster from someone who has already spent years understanding a subject. I call that liquid knowledge: knowledge that flows from someone's years of experience into a form another person can use quickly.

Transformer

Transformer is the architecture described above. In practical terms, it reduced the gap between people and machines. To give instructions to a computer, we used to need its language, first binary and assembly, then C, Java and Python. With transformers, you can give instructions in natural language and the machine responds in natural language.

That doesn't make programming useless. How clearly you write the instruction still decides the quality of the result, the same way the quality of your code decides what a program does, which is why prompting is a skill. And for engineering roles, you still need to program, because the model is an assistant inside a system you build, not the whole system.

What happens when you send a prompt

Here is the full path of one request, step by step.

1. Tokenization: your text is split into tokens, which are words, parts of words, spaces or punctuation, and each token is mapped to an ID from the model's vocabulary. The tokens page explains this in detail.
2. Embedding: each token ID is turned into a vector, a long list of numbers, from a table the model learned during training.
3. Transformer layers: the vectors pass through many layers. In each layer, attention lets every token gather information from the other tokens, and a feed-forward network transforms the result. After many layers, each vector carries the meaning of its token in this particular context.
4. Prediction: from the final vector, the model calculates a score for every token in its vocabulary, often 100000 or more, and turns those scores into probabilities that add up to 1.
5. Sampling: one token is picked from those probabilities. With a low temperature, the most likely one is almost always picked, and with a higher temperature, less likely tokens get a chance. The sampling page covers this.
6. Repeat: the new token is added to the end of the input, and the whole process runs again to predict the token after it. This continues until the model produces a special end token or hits the maximum length you set.

That last step is why you see answers appear word by word. The model doesn't write the answer and then show it to you, it generates it one token at a time, and each new token depends on everything before it, including the tokens it just generated.

Next-token prediction: the sky is blue

Take the sentence "The sky is". As a person, you can guess the next word, because you have read and heard English your whole life. It could be "clear", "cloudy" or "dark", but "blue" fits best.

An LLM does the same thing with probabilities. It might give "blue" the highest probability, "clear" a little less, and thousands of other tokens very small probabilities, and then sample from that distribution.

When you look at these values in an API that returns them, you will often see negative numbers next to each token. Those are log probabilities, the logarithm of the probability, which models use because multiplying many small probabilities together quickly becomes too small for a computer to store accurately. A log probability close to 0 means a probability close to 1, so the least negative number is the most likely token.

This is why people call LLMs a very powerful autocomplete. The phone keyboard that suggests your next word does a much simpler version of the same thing. The comparison is useful for remembering the mechanism, but it undersells what happens when a model with billions of parameters, trained on trillions of tokens, does it: to predict the next token well across so much text, it has to learn grammar, facts, reasoning patterns and styles of writing.

Attention, explained with a sentence

Consider these two sentences: "Himanshu is conducting a workshop. He will answer your questions." You know immediately that "he" refers to Himanshu. There is a strong relationship between those two words, and weaker relationships between "he" and the other words.

Attention is how the model learns those relationships. For every token, it calculates how much attention to pay to every other token, and builds an updated representation of that token from the ones that matter most.

People read in a similar way. When you read a paragraph and someone asks you about it later, you don't remember it word by word, you remember the key points.

There is a good everyday example of focusing on what matters. If you go to a vegetable seller and say "My mom gave me money to buy potatoes, and if I don't bring them home she will be angry", the seller ignores the whole story and hears one thing: potatoes. Understanding what a person wants, their intent, and the meaning around their words, their context, is what attention helps a model do.

Parameters: what the model knows

When a model is trained, it doesn't store the training text. It adjusts its parameters, the weights and biases of the neural network, so that its predictions get better.

Parameters are large matrices of numbers, and together they hold everything the model learned: grammar, relationships between words, facts, patterns and how important different words are in different contexts. You can think of them as the model's memory, but a compressed, statistical memory, closer to how you remember what you understood from a subject than to a copy of the textbook.

GPT-3 had 175 billion parameters, and today's models range from under a billion to hundreds of billions or more. When you see a model name like 8B or 70B, the B is billions of parameters. More parameters usually means more knowledge and ability, but also more memory, more compute and more cost to run, which is why small models are often the right choice for narrow tasks, as the model families page explains.

How an LLM is trained

Modern chat models go through three main stages.

Pre-training

The model reads a huge amount of text, from websites, books, code and other sources, and learns to predict the next token. GPT-3's training data came from five main sources, including a filtered version of Common Crawl (a large crawl of the web), books and Wikipedia, and open datasets like FineWeb today contain around 15 trillion tokens of filtered web text.

The result is a base model, sometimes called a foundation model. It understands language very well, but it doesn't know what you want from it. Ask it a question and it might continue with more questions, because on the internet, a question is often followed by other questions.

Instruction tuning

Next, the base model is trained further on examples of instructions and good responses, questions with answers, requests with completions, written or reviewed by people. This is supervised fine-tuning, and it teaches the model to behave like an assistant: to answer the question you asked.

Preference tuning

Finally, the model learns from feedback on which responses people prefer. The common method is reinforcement learning from human feedback (RLHF): people compare pairs of responses, a reward model learns those preferences, and the LLM is tuned to produce responses the reward model scores highly. Newer methods like direct preference optimization (DPO) reach a similar result more simply.

When ChatGPT occasionally shows you two answers and asks which one you prefer, you are giving exactly this kind of feedback.

Why the model doesn't learn from your chats

People are often surprised that a model used by millions of people doesn't get smarter with each conversation.

The reason is that a model's knowledge lives in its parameters, and those are fixed when training ends. When you chat, the model reads your messages and generates a response using its fixed parameters, but nothing in the conversation changes them.

Updating the model from every conversation isn't practical, for two reasons. Training needs huge amounts of compute, and it can't run continuously for millions of users at once. And the conversations are not clean training data: they are random questions, mistakes and personal information, and training on them directly would make the model worse, not better.

This has a few consequences you will see every day.

• Every model has a knowledge cutoff, the date its training data ends, and it knows nothing after that unless you give it the information.
• A new chat starts fresh. It is a new session, and the model doesn't know what you discussed in another chat.
• Memory features in apps like ChatGPT work by saving notes from your conversations and adding them to the prompt later, not by changing the model.
• If you want a model to use information it wasn't trained on, you put that information into the prompt, which is what retrieval-augmented generation (RAG) does.

How an LLM is different from a search engine

A search engine finds documents that already exist. When you search, it matches your query against an index of web pages, using keywords and also the context and intent of your query, and returns links to what it found.

An LLM generates new text from its parameters. It doesn't look anything up unless the application around it gives it a search tool or retrieved documents. That is why it can answer questions no web page answers directly, and also why it can produce a fluent answer that is wrong.

Retrieval-augmented generation combines the two: a search step finds relevant documents, and the LLM writes the answer from them.

How an LLM is different from an agent

An LLM takes a prompt and generates a response: one call, one answer.

An agent uses an LLM in a loop. It thinks about what to do, takes an action like calling a tool, observes the result, and thinks again, repeating until the task is done.

Ask an LLM what the weather is, and it can only guess or say it doesn't know. Ask an agent, and it calls a weather API and tells you the actual temperature. The LLM predicts, and the agent executes.

What this explains in practice

Once you see an LLM as a next-token predictor with fixed parameters, many everyday problems make sense.

What you see	Why it happens	Where to read more
The model forgets the start of a long conversation	It only sees what fits in its context window	Context windows
The same prompt gives different answers	A token is sampled from probabilities each time	Sampling
It miscounts letters in a word	It reads tokens, not letters	Tokens
It doesn't know recent events	Its parameters stop at the knowledge cutoff	Retrieval and RAG
It states something false confidently	It generates likely text, not verified text	Hallucination
Long chats cost more	Every turn resends the whole history	Pricing math

From the floor

In a workshop on how LLMs work, one builder asked whether a new chat in ChatGPT would remember what they had discussed in an earlier chat. I explained it with sessions in web development: when you open a website, a session starts, and when you close it, that session ends, and a new chat is a new session that starts fresh.

In the same workshop, another builder asked a sharp question: if LLMs are neural networks, and neural networks are inspired by how the brain learns, why doesn't the model learn from all the questions people ask it? The answer is the two reasons on this page, the compute it would take and the quality of the data, and it led us to RAG, which is how we give a model new information without training it again.

Related

• Tokens and tokenization
• Sampling: temperature, top-p and top-k
• Context windows
• Hallucination

Sources

• Attention Is All You Need, Vaswani and others (2017)
• Language Models are Few-Shot Learners (GPT-3), Brown and others (2020)
• Training language models to follow instructions with human feedback (InstructGPT), Ouyang and others (2022)
• FineWeb dataset, Hugging Face
• Introducing ChatGPT, OpenAI (November 2022)