How much VRAM do you need to run an LLM?

For inference, the memory you need is roughly the weights plus the KV cache plus runtime overhead. Weights take the parameter count times the bytes per parameter: about 2 GB per billion parameters at BF16, 1 GB at FP8 and roughly 0.5 GB at 4-bit. The KV cache grows with every token held in memory, across all concurrent requests, and at the long contexts agents need it rivals the weights. Serving frameworks do not hand the whole card to the model, so leave headroom before you pick a GPU. Without a GPU, the same arithmetic applies to system RAM, at lower speed.

• Weights = parameters × bytes per parameter (2 at BF16, 1 at FP8, about 0.5 at 4-bit).
• KV cache per token = 2 × layers × KV heads × head size × bytes; multiply by tokens held at once.
• Estimates differ mostly in the context length they assume; check it before comparing two numbers.
• Mixture-of-experts models must load every expert, even though few are active per token.
• Full fine-tuning with Adam needs about 16 bytes per parameter before activations; adapters need far less.

Serving a language model uses GPU memory for three things: the weights, the KV cache that holds the attention keys and values of every token in flight, and working memory for activations, the CUDA context and the serving framework itself.

The first two follow from the model card and your settings. The third depends on the runtime, and it is why a card is never fully available to the model. vLLM, for example, lets the model executor use a fixed fraction of GPU memory, set by --gpu-memory-utilization, which defaults to 0.92 in the current documentation [1], and by default it captures CUDA graphs, which “take up extra memory in the GPU” [2].

Weights need one value per parameter. Multiply the parameter count by the bytes per value: a billion parameters at BF16 take about 2 GB.

Qwen3-8B has 8.2 billion parameters [3], so its weights take about 16.4 GB at BF16, 8.2 GB at FP8 and roughly 4.1 GB at 4-bit. Quantised formats store scales next to the values, so real 4-bit files are slightly larger than half a byte per parameter.

Mixture-of-experts models load every expert

A mixture-of-experts model uses only some parameters per token, but all of them must be in memory. Qwen3-30B-A3B has 30.5 billion parameters in total and 3.3 billion activated [4]: it computes like a small model but needs about 61.0 GB for BF16 weights. The format matters as much as the count: OpenAI’s gpt-oss-120b has 117 billion parameters, and because its expert weights ship in MXFP4 it runs on a single 80 GB GPU, while gpt-oss-20b runs within 16 GB [5].

For every token the model is working on, each layer keeps a key vector and a value vector per KV head. The PagedAttention paper gives the example of OPT-13B, where one token takes 800 KB: 2 (key and value) × 5,120 (hidden size) × 40 (layers) × 2 bytes [6].

Newer models use grouped-query attention, with fewer KV heads than query heads, which shrinks the cache. Qwen3-8B has 32 query heads but 8 KV heads [3]. With the layer count and head size from each model’s configuration file, the cache at BF16 comes out as follows:

The token count that matters is everything held at once: context length × concurrent requests. Ten chats with 8K tokens each need the same cache as one chat with 80K tokens. That is why a model that “fits” in a quick test can run out of memory under load.

Two settings bring the cache down. vLLM notes that storing the KV cache in FP8 “can significantly reduce its memory footprint”, so more tokens fit in the same memory [11], and capping context length (max_model_len) and batch size (max_num_seqs) limits how much cache can be requested [2].

Adding weights and a 32,768-token KV cache at BF16 gives the memory to plan for before runtime overhead:

• Qwen3-8B needs about 21.2 GB at BF16: tight on a 24 GB card once overhead is added, comfortable on 48 GB, and about 13.0 GB with FP8 weights.
• Qwen3-30B-A3B computes like a 3B model but needs about 64.2 GB at BF16, because every expert stays loaded.
• Qwen2.5-72B-Instruct needs about 156.1 GB at BF16, more than two 80 GB GPUs leave for vLLM at its default setting (147.2 GB). With FP8 weights (83.4 GB) it fits on one 141 GB H200 or across two 80 GB GPUs. At 4-bit it fits a 48 GB card only with a shorter budget: about 39.0 GB with 8K tokens.

Why estimates for the same model differ

Memory calculators rarely disagree about the weights. They disagree about the context they assume, whether they count the KV cache at all, and how much of the card they treat as usable. The same model and precision give very different answers:

Before you compare two estimates, check the context length and the number of concurrent requests behind each. To run the arithmetic for your own model, precision and context, use the VRAM / model fit tool.

Agents need long contexts, and the KV cache grows with every token. Hermes Agent, for example, requires a model with at least 64,000 tokens of context and asks you to set a local model to 64K, for instance --ctx-size 65536 in llama.cpp [12]. Ollama uses a context window of 4,096 tokens unless you raise it with OLLAMA_CONTEXT_LENGTH or num_ctx [13], so the default is far below what such an agent needs.

At 65,536 tokens, Qwen3-8B needs about 26.1 GB with BF16 weights and cache, more than a 24 GB card holds. With FP8 weights and an FP8 KV cache it comes to about 13.0 GB. The model supports 32,768 tokens natively, so a 64K window also means enabling YaRN scaling as its model card describes [3].

Concurrency multiplies the cache again: in Ollama, the memory a model needs scales with the number of parallel requests times the context length [13]. Size for the number of conversations the agent runs at once, not for one.

vLLM’s own guidance for choosing a layout is short:

• If the model fits on a single GPU, run it on that GPU; distributed inference is probably unnecessary.
• If it is too large for one GPU but fits on one node with several GPUs, use tensor parallelism.
• If it is too large for one node, combine tensor parallelism with pipeline parallelism [14].

Tensor parallelism splits the model across GPUs (tensor_parallel_size) [2], and those GPUs then exchange data on every layer. That is where the interconnect in the GPU cloud checklist matters: GPUs linked by NVLink handle it far better than cards that only talk over PCIe.

Before adding GPUs, try the cheaper levers: quantise the weights, store the KV cache in FP8, and cap the context length and concurrency to what your application needs.

Training keeps far more than the weights. The ZeRO paper breaks down mixed-precision training with the Adam optimiser: 2 bytes per parameter for FP16 weights, 2 for gradients and 12 for the FP32 copy of the weights, momentum and variance, 16 bytes in total. A 1.5-billion-parameter GPT-2 therefore needs at least 24 GB, against 3 GB for its FP16 weights alone [15]. By the same arithmetic, fully fine-tuning Qwen3-8B needs about 131.2 GB for model states before any activations, which means several GPUs and a sharded optimiser.

Adapters change the budget

Low-rank adapters train a small number of extra weights instead. Compared with GPT-3 175B fine-tuned with Adam, the LoRA authors report 10,000 times fewer trainable parameters and a 3 times smaller GPU memory requirement [16]. QLoRA combines a 4-bit base model with adapters and made it possible to fine-tune a 65B model on a single 48 GB GPU [17].

With an estimate in hand, pick the smallest memory tier that leaves headroom for overhead and traffic peaks. The table lists every GPU in our catalog that has a verified price today:

Providers for each tier are listed in the inference hosting and GPU cloud comparisons. By memory tier: RTX 4090 cloud (24 GB), L40S hosting (48 GB) and H100 hosting (80 GB).

A model can also run from ordinary system memory. llama.cpp runs on x86 CPUs with AVX, AVX2, AVX512 and AMX, and its CPU+GPU hybrid mode exists to “partially accelerate models larger than the total VRAM capacity” [18]. Ollama shows where a loaded model lives: ollama ps reports 100% GPU, 100% CPU (system memory) or a split between the two [13].

The arithmetic does not change: weights plus KV cache must fit, now in RAM, next to the operating system and everything else on the server. On a 16 GB machine, Qwen3-8B at 4-bit with a 32,768-token cache needs about 8.9 GB and fits, while Qwen3-30B-A3B at 4-bit needs about 15.3 GB for its weights alone and does not leave enough room.

Expect CPU-only generation to be slower than on a GPU; that gap is what the hybrid mode above partially closes. It suits small models, light traffic and background agents; test your model at the context length you need before you pick a plan. CPU servers are compared under AI VPS, and providers with Ollama templates or guides under Ollama hosting.