AI glossary

A multi-step task orchestrated by an agent — plan → tool call → observe → iterate — as opposed to a single prompt-response.

A software component that plans, calls tools, observes outcomes, and continues acting toward a goal, using an LLM as its reasoning engine.

The maximum number of tokens an LLM can attend to in a single forward pass. Modern models range from 8K to 10M tokens.

A model architecture that routes each token to a subset of specialist sub-networks. Increases total parameters without proportionally increasing inference cost.

See MoE. Model where a routing network sends each token to a small subset of specialist sub-networks.

The chunks (typically 1-4 characters or ~0.75 word) that LLMs read and generate. Model input/output limits, pricing, and context windows are all measured in tokens.

A framework for running standardised benchmark suites against a model to score capability, safety, and regression.

A parameter-efficient fine-tuning method that freezes the base model and trains small rank-decomposed matrices, cutting VRAM needs by 10-100x versus full fine-tuning.

Grouping multiple requests into one forward pass to keep GPU utilisation high. Continuous batching lets requests join and leave mid-generation.

The cached key/value tensors from previous tokens that let a transformer generate each new token without recomputing the entire prompt. Memory-hungry at long contexts.

Time from request to response. For LLMs, split into time-to-first-token (TTFT) and time-per-output-token (TPOT).

Reducing the numeric precision of model weights (fp16 → int8 → int4) to shrink memory footprint and speed up inference with minimal quality loss.

An inference speedup where a small draft model proposes several tokens ahead; a large model verifies them in parallel. Trades a bit of compute for ~2x throughput.

Tokens or requests processed per second by an inference server. Trade-off with latency: larger batches = higher throughput but higher per-request latency.

The delay between sending a prompt and receiving the first generated token. Critical for interactive UX; grows with prompt length.

A prompting technique where the model is asked to reason step-by-step before answering, often producing dramatically better results on math/logic tasks.

A model mode (Claude, o-series) that spends visible internal reasoning budget before answering. Trades latency for depth on hard problems.

Providing the model with 2-10 example input-output pairs in the prompt to demonstrate the desired task pattern before asking it to complete a new instance.

The practice of shaping the input to an LLM — instructions, examples, format constraints — to elicit reliable output.

Asking the model to perform a task with no in-context examples, relying entirely on instruction-following ability from training.

Constraining a model's output to reference specific source documents, retrieved evidence, or a controlled knowledge base — the primary defense against hallucination.

When an LLM confidently states facts that are false or fabricates citations. The gap between output plausibility and output correctness.

A model feature that constrains generation to a supplied JSON schema, guaranteeing parseable, typed outputs.

See JSON mode. A guaranteed-schema output feature critical for agent pipelines that need machine-parseable results.

Algorithms (HNSW, IVF, ScaNN) that trade a tiny amount of retrieval accuracy for orders-of-magnitude speedups over brute-force nearest-neighbour search.

A retrieval model that embeds queries and documents independently. Fast (embeddings can be precomputed) but less accurate than cross-encoders.

A retrieval model that processes query and document jointly for each candidate. Very accurate but slow — used as the second stage in rerankers.

A dense vector representation of text (or image/audio) that captures semantic meaning. Similar meanings sit close together in vector space.

Combining vector (semantic) similarity with traditional keyword search (BM25) — vector search finds meaning-similar docs, keyword search catches exact-match queries the embeddings miss.

A second-stage retrieval step where a smaller cross-encoder scores each candidate document against the query, boosting precision above vector similarity alone.

A pattern where an LLM is given relevant documents (retrieved from a vector or keyword index) at query time, so its response is grounded in real data rather than only what it memorised during training.

Search where the query and documents are compared by embedding similarity rather than keyword overlap. Understands meaning across paraphrases.

A database optimised for storing embeddings and performing approximate-nearest-neighbour (ANN) search over millions to billions of vectors.

Umbrella term for research and engineering practices that reduce the risk of AI systems producing harmful, deceptive, or unaligned outputs.

The practice of shaping a model's behaviour to match human intent and values — via RLHF, DPO, constitutional AI, or filtered training data.

Runtime filters, classifiers, or policy checks that sit around an LLM to block harmful inputs/outputs or enforce topical scope.

A prompt attack that gets a model to bypass its safety training and produce content it was trained to refuse.

A sampling parameter (0 to 2) that controls output randomness. Lower = more deterministic; higher = more creative/varied. 0 is nearly greedy decoding.

A sampling constraint that considers only the k highest-probability tokens at each step. Often combined with top-p or temperature.

A sampling constraint that considers only the smallest set of tokens whose cumulative probability exceeds p (e.g. 0.9). Reduces low-probability outliers.

LLM-native mechanism where the model outputs a structured request to invoke a named tool with typed arguments, then integrates the tool's response into its final answer.

An open protocol standardising how LLM applications discover and invoke external tools, resources, and prompts. Introduced by Anthropic in 2024.

The pattern where an LLM is given a set of callable tools (search, code execution, database query) and can decide which to invoke as part of answering.

Small learned modules inserted into a frozen base model, allowing task-specific behaviour without touching the original weights.

A simpler alternative to RLHF that trains directly on preference pairs without a separate reward model. Widely adopted post-2024.

Continuing to train a pretrained base model on a smaller, task-specific dataset to specialise its behaviour.

Training a smaller student model to match the outputs of a larger teacher model, retaining much of the quality at a fraction of the cost.

Fine-tuning method that merges the SFT and preference-optimisation stages, further simplifying the DPO pipeline.

Umbrella term for fine-tuning approaches (LoRA, adapters, prefix tuning) that train a small fraction of a model's parameters.

A training approach where human preferences over pairs of model outputs are used to train a reward model, which then guides RL fine-tuning to align model behaviour.