How Much GPU VRAM Do You Need to Run an LLM?

A model needs roughly 2 GB of GPU memory per billion parameters at 16-bit precision, plus room for the KV cache and runtime overhead. An 8B model fits on one 24 GB card for a single short chat. A 70B model at 16-bit needs about 140 GB for the weights alone, which means at least two 80 GB GPUs for the weights and more once you add cache and overhead, or a quantized version.

That rule of thumb gets you close. This guide breaks the real number into three parts, so you can size a GPU for your own model, context length and number of users:

VRAM ≈ weights + KV cache + overhead

Figures are as of October 2026.

1. Model weights: parameters times bytes

Weights are the biggest and most predictable part. Multiply the parameter count by the bytes each parameter takes at your chosen precision:

weights (GB) = parameters (billions) × bytes per parameter

Inference almost never uses FP32. BF16 is the usual default, and 8-bit or 4-bit quantization trades a little quality for a much smaller footprint. 4-bit formats sit slightly above 0.5 bytes because they also store scaling values.

Mixture-of-experts (MoE) models need care. Only a few experts run per token, but all of them must sit in memory. Size an MoE model by its total parameters, not its active ones: a model with 671B total and 37B active parameters still needs memory for 671B.

2. KV cache: context length times users

The KV cache stores attention keys and values for every token in every active conversation. It grows with context length and with the number of requests running at once, so it is the part people forget:

KV cache (bytes) = 2 × layers × KV heads × head dim × bytes per value × tokens × concurrent sequences

The layer, KV-head and head-dimension values are in the model's config.json (num_hidden_layers, num_key_value_heads, and either head_dim or hidden_size divided by num_attention_heads). The leading 2 counts one key and one value.

Two worked examples at 16-bit cache precision. The Llama 3.3 70B figures match its published config: 80 layers, 8 KV heads, head dimension 128.

Those figures are for one conversation. Ten users each holding 8K tokens on the 70B model need about 27 GB of cache on top of the weights. Long roleplay chats and agent sessions are where this adds up fastest.

An 8-bit KV cache halves these numbers, at a small cost in quality on some models. The formula also assumes standard attention with grouped KV heads. Models that compress the cache, such as those using multi-head latent attention (DeepSeek's, for example) or sliding-window attention, need far less, so read their config before you apply it.

3. Overhead, and two full examples

The CUDA context, activations and the serving framework's buffers take the rest. A common rule of thumb is 10–20% on top of weights plus cache, and never less than 1–2 GB per GPU. Prompt-heavy workloads push toward the top of that range. Both examples below use 15%.

There is also a limit on how much of the card you can use. vLLM, for example, lets its engine use 90% of GPU memory by default (gpu_memory_utilization=0.9), and that budget covers weights, KV cache and activations together. It fills whatever is left after weights with cache, so a running server that shows the GPU almost full is normal. For sizing, the test is simple: your total has to fit inside 90% of the card's memory.

Example A: Llama 3.1 8B, 8K context, 4 concurrent chats

In BF16 this does not fit a 24 GB card at the default setting, because 90% of 24 GB is about 21.6 GB. It fits comfortably on a 48 GB card (about 43 GB usable). With 8-bit weights the total drops to about 14 GB, which fits a 24 GB card with about 7 GB to spare for more users.

Example B: Llama 3.3 70B, 8K context, 10 concurrent chats

• BF16 (about 192 GB): two 141 GB GPUs (about 254 GB usable) or four 80 GB GPUs (about 259 GB usable). Three 80 GB GPUs would have enough memory, but tensor parallelism needs a GPU count that divides the model's attention heads. vLLM stops with the error "Total number of attention heads must be divisible by tensor parallel size" otherwise. Llama 3.3 70B has 64 heads, so use 2, 4 or 8 GPUs, not 3.
• FP8 (about 112 GB): two 80 GB GPUs (about 144 GB usable) are enough.
• 4-bit (about 77 GB): this does not fit one 80 GB GPU, which has about 72 GB usable. Serve fewer users, or switch the KV cache to 8-bit, which halves the cache to about 13 GB and brings the total to about 61 GB. A 141 GB GPU leaves plenty of room either way.

Splitting a model across GPUs adds a little overhead on each card, so keep some headroom.

4. Which GPU fits which model

This table applies the formulas above to common data-center GPUs. The ceilings are rough, not benchmarks. They assume 90% of the card is usable, 15% overhead, 8K context and about four concurrent users, and they are for dense models. Treat them as a starting point, then run the numbers for your own model.

Memory sizes for the H200, B200 and B300 come from NVIDIA's HGX documentation.

Three checks before you pick:

• Long context or many users? Add the KV cache for your real context length and concurrency. On a busy server it can match the weights.
• One GPU or several? A model that fits on one GPU is simpler and avoids cross-GPU traffic. Split across GPUs only when it doesn't fit, and use a GPU count that divides the model's attention heads.
• Quality at lower precision? Quantization trades some quality for memory. Test FP8 and 4-bit on your own prompts and compare them with BF16 before relying on either.

5. Running it on Nebula Block

You only need to size a GPU if you host the model yourself. Nebula Block has two ways to run a model, and the first one skips sizing entirely.

For serverless, check the model catalog first. If your model is already there, you don't need a GPU at all.

For GPU instances, work out your total in sections 1 to 3, then choose a configuration whose usable memory (90% of the total across its GPUs) covers it with about 10% headroom. A few facts from the docs help with that:

• Hardware and regions. Nebula Block rents B300, B200, H200, H100, A100, L40/L40S and RTX GPUs, in configurations from a single GPU up to 8-GPU nodes (10 in some regions), across Canada, the United States, Finland, France and Norway.
• Checking what's in stock. The Deploy page in the console and the List Products API show current availability. The API returns the GPU model and gpu_count for each configuration, so match them against the memory sizes in section 4. Check is_available rather than stock.
• Access and limits. GPU instances need Tier 3, which is a $10 deposit. An hourly spending limit caps how much you can run at once, so multi-GPU nodes can need a higher limit than a new account starts with.
• Billing. Instances are billed hourly for as long as they exist, and stopping an instance does not stop billing. Delete instances you are finished with.

Conclusion

Start with 2 GB per billion parameters in BF16, then add the KV cache for your real context length and user count, plus about 15% overhead. Check the total against 90% of the card's memory, not 100%. If it doesn't fit, quantize to FP8 or 4-bit, serve fewer users or move to a bigger card, and use a GPU count that divides the model's attention heads when you split it. If the model is already in a serverless catalog, you may not need to size anything at all.

Learn more at

• Email: contact@nebulablock.com
• Website: nebulablock.com
• Docs: docs.nebulablock.com
• Book a call: nebulablock.com/contact