LLM
A large language model (LLM) is a neural network with billions of parameters, trained on internet-scale text to do one deceptively simple thing: predict the next token given the tokens so far. Scaled up far enough, that single objective produces systems that write fluent prose, answer questions, generate working code, translate languages, and follow instructions — capabilities nobody explicitly programmed in. GPT, Claude, Llama, and Gemini are all LLMs. The diagram traces what actually happens between a prompt going in and a word coming out.
Everything is next-token prediction. During training the model sees enormous amounts of text with the next word hidden, and it adjusts its weights to raise the probability it would have assigned to the real next token. There is no separate "reasoning module" or "fact database" — grammar, world knowledge, translation, and arithmetic are all compressed into the weights as a side effect of getting good at this one guessing game.
The transformer block is the repeating unit. Each layer has two parts: a self-attention step, where every token looks at the others and pulls in the context it needs, and a feed-forward network that processes each position independently. Stacking dozens to over a hundred of these blocks lets early layers capture surface patterns and later layers capture meaning, syntax, and long-range structure.
Scale is the defining property. LLMs are distinguished from earlier language models by sheer size — parameters, training tokens, and compute. Empirical scaling laws show loss falling predictably as all three grow together, and certain abilities (in-context learning, multi-step reasoning) appear only past a size threshold. This predictability is why labs are willing to spend enormous sums on a single training run.
Pretraining teaches language; post-training teaches behavior. A raw pretrained model is a talented autocomplete engine but not yet a helpful assistant. A second stage — instruction tuning on curated examples, then reinforcement learning from human feedback (RLHF) — aligns it to follow instructions, stay on task, and refuse harmful requests. Most of the "personality" of a deployed chatbot comes from this phase, not pretraining.
Inference is autoregressive. To answer, the model generates one token, appends it to the input, and runs again — looping until it emits a stop token. Each step reuses cached attention state (the KV cache) so it does not recompute the whole history, which is why the first token is slow (prefill) and later tokens are fast (decode).
| Component | Role | Analogy |
|---|---|---|
| Tokenizer | splits text into subword tokens | breaking a sentence into Lego pieces |
| Embeddings | turn token IDs into vectors | giving each piece coordinates in meaning-space |
| Attention | tokens share context | everyone in the room comparing notes |
| Feed-forward | per-token processing | each token thinking on its own |
| Unembedding | vectors back to token scores | scoring every possible next word |
Read an LLM through a *next-token-prediction* lens rather than a *knowledge-database* lens: it does not look facts up, it reconstructs the most probable continuation from patterns compressed into its weights during training. That single framing explains its strengths — fluency, generalization, in-context learning — and its failure modes — confident hallucination, sensitivity to phrasing, and knowledge frozen at its training cutoff — because all of them fall out of a system optimized to predict text rather than to store truth.