Estimating What Fits in Your VRAM — Quantization and Memory Math for Local LLMs

You downloaded a model. You started it. It died. The error says out of memory, but not how many gigabytes it wanted. The filename carries a tag like Q4_K_M, and next to it in the same folder sit Q5_K_M and Q8_0, with nothing to tell you which one belongs on your card.

The first wall in local LLM work is not quality. It is this estimate. The good news is that the dominant term is a single multiplication. This article covers that formula, and how to read the numbers you feed into it. Everything quoted here comes from published measurements; anything I derived is labelled as an estimate.

Almost every failure to start is a memory failure

Three things add up to the requirement:

• The model weights, claimed the moment it starts and never released.
• The KV cache, which grows with the length of the conversation.
• Runtime overhead — scratch space the framework allocates for intermediate values and for handling concurrent requests.

On the third: llama-server takes –parallel N to set the number of server slots, defaulting to auto. Accept more simultaneous requests and the working set grows with them. If you are standing the thing up as a shared server, budget for it.

The first item dominates, and it is the one you can calculate exactly in advance. The order also matters. Weights are allocated up front, so if they do not fit the process dies immediately — an obvious failure. The second is the nasty one, because it starts fine and dies later, after you have already concluded the configuration is correct. The third varies by environment and is better absorbed with headroom than modelled precisely.

The size of the weights is one multiplication

Bytes required = parameter count x bits per parameter / 8.

That is the whole thing. Check it against published measurements. Llama-3.1-8B quantised to Q4_K_M is listed at 4.8944 bits per parameter and 4.58 GiB. Put that in: 8.03e9 x 4.8944 / 8 = 4.913e9 bytes, or 4.575 GiB. It matches.

Try Q8_0 as well: 8.5008 bits, published size 7.95 GiB. The formula gives 8.03e9 x 8.5008 / 8 = 8.532e9 bytes, 7.945 GiB. Also a match. The formula can be trusted.

Reading the quantisation label

Q4_K_M decomposes:

• Q4 — the base bit width. Four bits.
• _K — the K-quant scheme, which keeps a scale per block to preserve accuracy.
• _M — the size variant within that family. S, M and L, small to large.

The trap is that Q4 does not mean four bits in practice. The published figure for Q4_K_M is 4.8944 bits, closer to five, because scales and per-block correction data ride along. Take the number in the filename literally and you will underestimate by about twenty percent.

Keep the published table within reach

These are the figures llama.cpp publishes for Llama-3.1-8B, quoted as-is.

Two things fall out of it. Dropping from F16 to Q4_K_M cuts the file to roughly a third — 14.96 GiB becomes 4.58 GiB, which is the difference between fitting and not. And the gap between the Q4 and Q5 families is under a gigabyte on an 8B model, which is often affordable.

Applying it to a different model

Qwen3-8B publishes 8.2B total parameters and 6.95B non-embedding. Estimating at Q4_K_M rates: 8.2e9 x 4.8944 / 8 = 5.017e9 bytes, about 4.67 GiB. This is an estimate. Embedding and output layers are frequently quantised at a different width, so the real file moves either side of it. It is still precise enough to answer the only question that matters at this stage: the weights fit on an 8 GB card.

The second figure is worth a moment too. The 1.25B gap between 8.2B total and 6.95B non-embedding is the embedding layer, which scales with vocabulary size — so it swells on models with broad multilingual coverage. If you need Japanese, that is not a layer you want to compress hard, which is exactly why quantisation tools offer to keep it at higher precision.

The formula does not care about scale. At around 1B parameters you are into single-board-computer territory, and how usable that really is was the subject of the Kimi K3 piece at the opposite end of the range — 2.8 trillion parameters, same arithmetic.

KV cache — where long conversations bite

The weights fit, and then it dies once the conversation gets long. That is the KV cache.

At every step the model holds intermediate representations for all preceding tokens. That store grows linearly with context length:

KV cache size is approximately 2 x layers x KV heads x head dimension x context length x bytes per element.

The leading 2 is Key and Value. Qwen3-8B has 36 layers. Head dimension is not on the model card, so I will not put a byte figure here — the structural point is the linear term. Qwen3-8B is native to 32,768 tokens and stretches to 131,072 with YaRN. Quadruple the context and you quadruple the cache.

GQA is doing a lot of the work

Grouped Query Attention is why this is survivable. The Qwen3-8B card lists attention heads as 32 for Q and 8 for KV. Cache size is set by the KV side, so that is a four-fold reduction against a design where every head carries its own KV.

Knowing this changes how you read a model card. If you want long context, look at the ratio of KV heads, not only the parameter count. Two models of the same size where one falls over on long documents is usually explained by that ratio. The architecture section of a model card is most useful for predicting memory behaviour, not quality.

Two things to cut before you drop the quantisation

When memory runs short, the reflex is to step down one quantisation level. There are two moves that come first, and neither touches quality directly.

First, reduce the context length. The cache scales linearly, so going from 32,768 to 8,192 quarters it. Jobs that genuinely need thirty thousand tokens of context are rarer than people assume, and sacrificing quality to reserve length you are not using has the order backwards.

Second, quantise the KV cache itself. Weights and cache take separate precision settings. Keeping the weights sharp and compressing only the cache is a more legible trade than shaving everything uniformly.

If both fail, then drop the quantisation. Fixing the order lets quality loss be the last resort rather than the first. There is a practical reason too: context length and cache precision are one line in a config and instantly reversible, whereas changing quantisation means downloading a different file. Try the reversible moves first — that is basic troubleshooting discipline.

If it still will not fit, you can offload part of the model to system memory. Speed drops sharply. Running and being usable are different claims.

Four runtimes, compared on what they document

If you are one person on one desktop, ease of installation wins. If several people share the box, pick something whose concurrency behaviour is written down.

OpenAI compatibility is the shared language

Line those four up and the common factor is obvious: all of them speak an OpenAI-compatible endpoint. The LM Studio docs put it plainly — change the base URL to http://localhost:1234/v1 and existing clients work. What that unlocks, including tool calling and MCP, is covered in wiring a local LLM into real work.

The estimate, as a procedure

1. Multiply: parameter count x bits per parameter / 8.
2. For a mixture-of-experts model, use the total parameters, not the active ones.
3. Give the remainder to the KV cache, and set context length deliberately rather than by default.

Step two is where people lose an order of magnitude. Qwen3.6-35B-A3B activates 3B parameters but you still have to load all 35B into memory. Confusing compute with capacity produces exactly the failure mode of assuming 3B is light, then watching it ask for 35B worth of memory.

Starting small is faster

One practical principle for choosing. Qwen3 ships dense models at 0.6B, 1.7B, 4B, 8B, 14B and 32B alongside 30B-A3B and 235B-A22B mixtures, so you can move up a rung at a time instead of guessing.

Better still, start with a model whose card states its requirement. gpt-oss-20b says it runs within 16GB of memory: 21B total, 3.6B active, Apache 2.0. A model with a published memory figure carries the least uncertainty as a first pick.

How that local cost compares against the cloud invoice is worked out in reading your cloud AI bill before you buy a GPU, and the hardware side is in what to buy first for local AI.

Summary — one multiplication and three additions

• Bytes required = parameter count x bits per parameter / 8, confirmed against two published measurements.
• Q4_K_M is really 4.8944 bits, about twenty percent above the 4 in its name.
• F16 to Q4_K_M cuts an 8B model from 14.96 GiB to 4.58 GiB.
• The KV cache scales linearly with context length; GQA is what keeps it affordable.
• Cut context length and cache precision before you cut quantisation — both are reversible in one line.
• For MoE models, memory follows total parameters, not active ones.
• A model card that publishes its memory requirement is the safest first pick.

Sources

• llama.cpp — quantize README (bits per parameter and file size by format)
• llama.cpp — llama-server README (OpenAI-compatible routes, –parallel)
• OpenAI — gpt-oss-20b model card (21B total, 3.6B active, runs within 16GB)
• Qwen — Qwen3.6-35B-A3B model card (35B total, 3B active MoE)

ブラウザだけでできる本格的なAI画像生成【ConoHa AI Canvas】

swiftwand

X

AIを使って、毎日の生活をもっと快適にするアイデアや将来像を発信しています。 初心者にもわかりやすく、すぐに取り入れられる実践的な情報をお届けします。 Sharing ideas and visions for a better daily life with AI. Practical tips that anyone can start using right away.