VRAM and model fit calculator

Formula

Weights

parameters × bits per weight ÷ 8

KV cache (full attention)

2 × layers × KV heads × head dimension × context × batch × bytes per value

Serving range

low = weights + KV cache + 1 GB; high = 1.2 × weights + KV cache + 1 GB

Fine-tuning range

low = model states + activations + 1 GB; high = 1.2 × (model states + activations) + 1 GB

Comfortable fit

high estimate ≤ 90 % of the GPU's listed memory

Assumptions

1. Bits per weight: BF16/FP16 16; FP8/INT8 8; 4-bit AWQ/GPTQ 4.16 (4-bit weights plus an FP16 scale and a 4-bit zero point per 128 weights); GGUF Q8_0 8.5, Q5_K_M 5.7 and Q4_K_M 4.89, as measured by llama.cpp. 8-bit and 4-bit formats keep the embeddings and output head at 16 bits. “As published” uses the dtypes of the model's own checkpoint, with MXFP4 at 4.25 bits.
2. The KV cache uses each model's own layer layout: sliding-window layers stop growing at the window, latent-attention layers cache one compressed latent per token, and linear-attention layers keep a fixed state per sequence. Keys and values are both cached, as in the reference implementations.
3. Runtime overhead is 1 GB per GPU for the CUDA context and runtime buffers. The high end adds 20 % on the weights for activations, temporary buffers and fragmentation, following EleutherAI's rule of thumb.
4. With several GPUs, weights and cache are split evenly (tensor parallelism) and the runtime overhead is counted on every GPU. The tool looks at up to 8 GPUs, one server.
5. Full fine-tuning counts 16 bytes per parameter: BF16 weights and gradients, FP32 master weights and two Adam moments. LoRA keeps BF16 weights frozen; QLoRA stores them in NF4 at 4.127 bits. Both train rank-16 adapters on the attention and dense MLP projections (attention only for mixture-of-experts models) at 16 bytes per adapter parameter.
6. Training activations assume full gradient checkpointing: 2 bytes per token, hidden unit and layer for the stored layer inputs, 34 bytes per token and hidden unit for the layer being recomputed, and FP32 logits. Model states are sharded across GPUs (FSDP or ZeRO-3); activations are per GPU.
7. A fit is comfortable when the high estimate uses at most 90 % of the GPU's listed memory (vLLM reserves 92 % by default) and tight when the low estimate fits in the listed memory. Vision encoders' activations, draft models for speculative decoding and CUDA graph buffers are not included.

References

• EleutherAI, Transformer Math 101 (opens in a new tab) ↗≤ 20 % inference overhead, AdamW memory, activation recomputation
• Rajbhandari et al., ZeRO (arXiv 1910.02054) (opens in a new tab) ↗16 bytes per parameter for mixed-precision Adam
• Dettmers et al., QLoRA (arXiv 2305.14314) (opens in a new tab) ↗NF4 with double quantisation, 0.127 bits of constants per parameter
• Lin et al., AWQ (arXiv 2306.00978) (opens in a new tab) ↗4-bit weight quantisation with group size 128
• llama.cpp, quantize README (opens in a new tab) ↗measured bits per weight of GGUF formats
• vLLM engine arguments (opens in a new tab) ↗default gpu_memory_utilization of 0.92
• Hermes Agent quickstart (opens in a new tab) ↗a local model for Hermes Agent needs at least 64,000 tokens of context (the 64K option)