The cached key/value tensors from previous tokens that let a transformer generate each new token without recomputing the entire prompt. Memory-hungry at long contexts.
The cached key/value tensors from previous tokens that let a transformer generate each new token without recomputing the entire prompt. Memory-hungry at long contexts.