Two transformer models with the same parameter count can have very different memory requirements during inference. A major reason is the structure of their attention layers. The number of key-value heads, the representation stored for each token, and the number of earlier tokens retained by each layer determine how much KV cache a request needs. […]