Serving an LLM costs more memory than the model file suggests. This breaks the total into weights, KV cache and runtime overhead for real architectures, and checks it against common GPUs. GB means GiB, as nvidia-smi reports it.
Memory grows linearly here; attention compute grows quadratically, which this figure doesn't show.
Weights are fixed by architecture and serving precision — a 7B model in
fp16 is ~14 GB no matter what you do at inference time. The KV cache is
the part everyone underestimates: for every token of every active sequence, the model
stores a key and a value vector per KV head, per layer, so it never recomputes attention
over the whole context. That's
2 × layers × kv_heads × head_dim × bytes per token —
multiplied by context length and by concurrent requests.
Overhead is a rough allowance for the CUDA context, allocator
fragmentation and activation buffers; an estimate, not a measurement.
Most recent models use grouped-query attention: many query heads share few KV heads, shrinking the cache without touching the weights. Llama 3 8B uses 8 KV heads; Phi-3 Mini 3.8B has no GQA at all — 32 full heads. Result: Phi-3's KV cache per token is 3× larger than Llama 3 8B's, despite having under half the parameters. Past a certain context length, the smaller model runs out of memory first. Switch between those two presets at 64K context and watch the cache overtake the weights.
Paged attention pools, CUDA graphs and multi-GPU sharding all change the picture and aren't modeled. Treat the total as a lower bound: if it doesn't fit here, it won't run; if it fits with room to spare, you're probably fine — the last few GB are framework-dependent.