What Large Language Models Actually Do When They Answer You

You type a question into a chat box, press enter, and two seconds later a tidy answer appears. It has paragraphs. It has bullet points. It addresses your exact wording, and it sounds like someone who knows what they are talking about. That impression is the most important thing to understand — and the easiest thing to be misled by. Underneath the polished surface, the system is not looking anything up. It is doing one narrow thing, over and over: guessing which chunk of text should come next.

Once you see that clearly, the strengths make sense. So do the failures.

The whole trick is next-word prediction

Your message is first chopped into tokens, which are pieces of words rather than whole words. Something like "unbelievable" might become "un", "believ", "able". A short question can be a dozen tokens; a pasted document can be thousands.

Then the model does its only job. It looks at every token so far and produces a probability for each possible next token in its vocabulary — tens of thousands of them. It picks one, appends it to the sequence, and runs the calculation again on the slightly longer sequence.

That loop is the entire generation process. Ask for a haiku, a SQL query or an apology email, and the mechanism is identical. The model is not retrieving a finished answer from a database; it is building one token by token, where each choice depends on everything before it.

Why such a simple loop looks like thinking

Because predicting text well is hard. To guess the next token accurately across a huge range of writing, a model has to absorb grammar, tone, formatting conventions, code syntax and a vast web of factual associations. "The capital of France is ___" has one overwhelmingly likely continuation. "She poured the milk into the ___" has several — glass, mug, bowl — and the right pick depends on context from sentences earlier.

Step-by-step explanations emerge for the same reason. If the training text usually walks through a calculation or a debugging session in order, a model that predicts that text well will produce plausible intermediate steps. That often resembles reasoning. It is still, mechanically, pattern completion.

Training data is a snapshot, not a live feed

During training, the model reads an enormous body of text and adjusts its internal parameters so the actual next tokens become more likely. What comes out is a compressed statistical impression of that text — not the text itself, and not an index you can search.

Two consequences matter in everyday use:

• It has a cutoff. Unless a search tool is connected, the model knows nothing about events, prices, releases or personnel changes after its training data ends.
• Coverage is uneven. Topics that appear constantly — mainstream programming languages, popular history, common medical vocabulary — are represented richly. Niche local details, smaller languages, recent regulation and specialist jargon are thin. Thin coverage is where confident errors cluster.

A later stage of training, often using curated examples and human feedback, shapes the raw predictor into something that answers politely and in the requested format. That is why it feels like an assistant. It also creates a bias toward giving some answer rather than admitting ignorance.

Confidence is a writing style, not a measurement

This is the part that trips people up. A model's certainty and its accuracy come from the same process, and the two are not linked. There is no separate step that checks a claim before it is spoken.

The tone is learned. Helpful explanatory writing tends to be declarative: "X is Y." So the model produces declarative sentences. When the right answer is common in the training text, this works well. When you ask about something rare — a page number, a middle name, a paper title, a clause in a local ordinance — a fluent wrong answer can be more statistically smooth than the correct one. Names, dates, numbers and citations are especially vulnerable, because invented details often fit the shape of real ones perfectly.

Ask for a source and you may get an author, a journal, a volume number and a year that all look right and none of which exist. The model is not lying to you. It is completing a pattern.

Where wrong answers actually come from

• Thin or ambiguous signal in the training text about a narrow subject.
• No live access, so anything time-sensitive may be stale or invented.
• Lossy compression. Detail blurs when the patterns of a huge corpus are squeezed into fixed parameters.
• Silent interpretation. Your question had three readings; it chose one without saying so.
• Agreement bias. It tends to accept the premise inside your question, including a false one.

How to get more reliable output

1. Supply the material. Paste the document, the error message, the dataset. Grounding beats general knowledge.
2. Specify the shape. "Three bullets, no preamble" or "a two-column table" removes guesswork.
3. Ask for uncertainty. "Flag anything you are not sure about" or "list what a reader should verify."
4. Ask for the opposite case. "Argue against this" surfaces weaknesses that a confident draft hides.
5. Separate generating from checking. Drafting and verifying are different jobs; never let one answer do both unchecked.
6. Verify the load-bearing claims. Names, numbers, quotes, dates and any legal or medical statement need a primary source you have opened yourself.

Treat the first answer as a draft

The most useful mental model is simple: you are working with a very fast, widely read writer who has never once said "I don't know" and never checks anything. For drafting, summarising, translating, rewriting code, brainstorming and decoding unfamiliar vocabulary, that is genuinely valuable. For facts that carry consequences, it is a starting point, not a verdict.

So read the reply and ask which claims the decision actually rests on, then confirm those separately. On health, money or legal matters, use the draft to prepare sharper questions for a qualified professional rather than as the final word. Then ask a follow-up. Ask it to criticise its own answer. The second pass is often noticeably better than the first — because the first was never anything more than a plausible continuation.

Photo: 1139623 / Pixabay