How LLMs work · for Product Owners

What happens when you press Enter

You use ChatGPT, Copilot or Claude every week. And sometimes it does things that make you wonder.

Press → or a clicker for the next step. P plays everything on its own.

Who has seen one of these?

Each of them has a reason, and we will meet all three on the way.

The first clue is what the model actually receives. On screen, you see a chat. Here is our message: a typical Product Owner request, with 40 pieces of feedback pasted in.

The model sees something else: one long document. The app’s instructions, what it remembers about you, the files and emails it found, everything said earlier, and your message.

And the whole document goes back in, every time you press Enter. Let’s follow it.

This is the route your message takes. The orange stops are where the context comes in, and the context decides much of what you get back.

Chapter 1 · Assembly

More than your message goes in

Let’s look at that document more closely. You typed one message.

The app adds its own instructions, which you never see, and what it remembers about you. Tools like Microsoft 365 Copilot also pull in emails, files and documents they consider relevant, often the largest part. Then everything said earlier in this chat.

All of it becomes one long document, and you typed only a small part of it. That document is all the model gets.

The word counts in this chapter are illustrative examples.

Try it

The model only sees one long document, and you wrote only part of it.

Chapter 2 · Tokens

Text becomes tokens

Before the model does anything with your message, the text is cut into small pieces. Each piece is called a token.

This is our message from before: 21 words.

A token is often a whole short word, sometimes only part of a longer one. Punctuation gets its own.

The dot marks a space. It belongs to the piece after it. This is how the GPT-2 tokenizer cuts our message: 25 tokens for 21 words.

Then every token is replaced by a number from a fixed list. The model receives these numbers, not text.

These are the real numbers GPT-2 uses. Other models use other lists, so the same piece gets another number.

Pieces do not have to be words. One reported example for the GPT-4o tokenizer: “strawberry” becomes st, raw and berry.

That is one reason letter counting goes wrong. It is not the whole story: the same research finds that models can recognise letters and still miscount.

A historical example, and current models may do better.

The same request in German is cut into more pieces. In this GPT-2 example: 12 tokens in English, 31 in German.

GPT-2 is old and English-centred, so this is an extreme case. One preprint measured about 1.76 tokens per German word on average, with six tokenizers.

For English, 100 tokens are about 75 words: roughly 1.3 tokens per word. German needs more, roughly 1.5 to 2 per word. That is an estimate.

Limits and prices are counted in tokens, not words. The same content can use more of the window, and cost more, in German.

English: OpenAI rule of thumb. German: one preprint’s average (Ovcharov 2026), 1.76 per word. Chapter 1 counted words to keep it simple.

Try it

Type anything and watch it being cut into tokens.

Show token numbers

GPT-2 tokenizer (2019, MIT licence). Current models use larger vocabularies, so their splits and counts differ. For real limits, use the tool’s own counter. Try “strawberry” with a space in front: GPT-2 cuts it differently.

Models read pieces of words, not letters. Limits and prices are counted in tokens.

Sources: OpenAI, understanding and counting tokens · Fu et al. 2024, letter counting · Ovcharov 2026, tokenizer tax (preprint, estimate) · GPT-2 tokenizer files, MIT licence. Accessed 19 October 2026.

Chapter 3 · Meaning

Tokens become meaning

A token is just a number. The model gives every number a position on a map of meaning.

Real models place each token with thousands of numbers, not two. This 2D map is an illustration, and every position on it is illustrative.

Words that are used in similar ways end up close together. Animals sit with animals, cities with cities, backlog with sprint.

Directions on the map carry meaning too. In early word-embedding research, the step from “man” to “woman” turned out to match the step from “king” to “queen”.

A classic illustration from that research. The positions here are illustrative.

Start at “king”, take the same step, and you land near “queen”: king − man + woman ≈ queen.

Meaning is not stored as a definition. It is a position, and distances and directions between positions carry it.

Search uses the same idea. Your question gets a position too, and the documents nearest to it are pulled in.

This is the idea behind how many tools choose the “relevant” files from chapter 1.

But near in meaning is not the same as right. The old FAQ looks relevant, so it is picked. The exit survey says the same thing in other words, so it is missed.

Both documents are invented examples for this map.

Try it

Similar meanings sit close together. Tools use the same idea to find “relevant” documents, and the closest document is not always the right one.

Sources: Mikolov, Yih & Zweig (2013), Linguistic Regularities in Continuous Space Word Representations (king − man + woman example). OpenAI, Vector embeddings guide (default 1,536 or 3,072 numbers per text for its embedding models). All positions, documents and the 2D map are illustrative. Accessed 19 October 2026.

Chapter 4 · Attention

Words look at each other

Now we are inside the model. Take this sentence: to make sense of “was”, the model has to work out who was sleeping.

Each word looks back at the words before it and decides which of them matter for it. Here “was” draws a thick line to “cat”: the cat is the one sleeping. The thin lines go to words that matter less.

The percentages are illustrative: hand-made to show the idea, not measured from a real model.

Every word does the same, one after another. Each looks only at the words that came before it, never ahead.

The model does not look just once. It runs several lenses side by side, called attention heads, and each one picks out a different relationship: who does what, when something happened, what a pronoun points to.

Illustrative: in a real model the lenses are learned, and they are rarely this tidy.

Word order is encoded too: every word carries its position, so “dog bites man” and “man bites dog” are not the same.

Now zoom out. Compare every word with every other word and you get a grid, one cell per comparison. Our eight words make 8 × 8 = 64 cells. The highlighted row belongs to “was”.

A real model skips the cells that would look ahead, about half of them. The growth below is the same either way.

Make the text longer and the grid grows with the square of its length: 10 words make 100 comparisons, 100 words make 10,000, and 1,000 words make 1,000,000.

That growth is one reason long context costs more time and money.

Try it

Pick a word and a lens, then stretch the text.

Each word decides which earlier words matter for it. That is how the model uses context, and why long context costs more.

Sources: 0xkato, How LLMs Actually Work (attention, multi-head attention, positional encoding, causal masking, quadratic cost). The attention weights and lenses are illustrative, made for this page. The comparison counts are plain arithmetic: words × words. Accessed 19 October 2026.

Chapter 8 · The round trip

You reply, and the whole conversation is back in play

Let’s follow a conversation over several turns. Turn one: you paste 40 pieces of feedback and ask for the main themes.

The app assembles the context, its own instructions plus your message, and the model works through all of it.

The answer comes back one token at a time and becomes part of the conversation.

You reply. The model kept nothing from turn one, so the whole conversation is supplied again.

The part it processed last time can come from a cache. It is not recomputed through every layer, but it still takes up space in the window.

With every turn the context grows. Each new word of the answer is computed with all of it in view.

Until the window is almost full.

Then the app condenses the oldest turns into a summary to make room. Their details are gone, and the cache starts over from that point.

Claude documents this automatic summarising. Other apps cut or summarise in their own way.

Try it

Reply, edit an earlier message, or start a new chat, and watch the context.

The model keeps no memory between answers. The conversation is the memory.

Sources: OpenAI, conversation state · Anthropic, Messages API · OpenAI, prompt caching · Claude, context window and summarising. Accessed 19 October 2026.