Throughput

Tokens or requests processed per second by an inference server. Trade-off with latency: larger batches = higher throughput but higher per-request latency.