What to Buy First for Local AI — Choosing a GPU by VRAM, Power and Noise

You have decided to run AI on your own hardware. The next question is what to buy, and the usual mistake is to start by comparing products. Sort the criteria first and most of the shortlist eliminates itself.

This article works through the order: capacity, then bandwidth, then power, then where the thing physically sits. All specifications quoted come from manufacturer pages.

Do not start from the wrong question

The question is not which GPU is fastest. It is whether the model you want to run will load at all. A card that cannot hold the weights does not run slowly; it does not run. Speed is a question you only get to ask after that.

So the first number is memory capacity, and the second is memory bandwidth. Compute throughput matters less than either for this workload.

VRAM capacity is the first constraint

Roughly what each capacity band opens up:

One caution on reading that table. Start from a configuration with about 1.5x headroom over the model size. On 8 GB, aim at a model around 5 GB; on 16 GB, around 10 GB. The remainder goes to the KV cache and working space, and the arithmetic behind that is in estimating what fits in your VRAM.

Current cards, capacity and power side by side

Read that on capacity alone and there is an interesting inversion. The 16 GB RTX 5060 Ti holds more than the 12 GB RTX 5070 above it. The 5070 is the stronger card on compute, but on the first gate — does it load — the 5060 Ti is the better one.

The same thing happens between the 4090 and the 5080. The 5080 is the newer generation; the 4090 has 24 GB against 16.

Note also that the 5060 Ti exists in 16 GB and 8 GB versions. The name does not tell you which, and for this workload the difference is decisive.

Bandwidth sets the generation speed

Once it loads, the pace is set by memory bandwidth. Producing each token requires reading the model weights, so the rate at which weights can be read is the ceiling. GDDR7 and GDDR6X are not the same here.

Going one level deeper, generation has two phases: reading the input, and emitting one token at a time. The second is bandwidth-bound. This is also why mixture-of-experts models feel faster than their size suggests — each token only reads the activated expert blocks, so the read volume is small relative to the total parameter count, even though all of it has to be resident.

Unified memory as the other answer

The advantage is plain: a large capacity treated as one pool. Reaching 96 GB on discrete GPUs means several cards, a case that fits them and a power supply that feeds them. The drawbacks are equally plain — it cannot be upgraded later, and raw compute trails a high-end discrete card.

The decision rule: if capacity is the binding constraint, unified memory is worth considering. If you need speed within a capacity a single card can already hold, a discrete GPU wins.

Working out the electricity

Power consumption is an ongoing cost, and easy to calculate. At the Japanese reference rate of 31 yen per kWh, running two hours a day for thirty days:

These are smaller than people expect, and that is the point: at two hours a day, electricity is not the deciding factor. It becomes one on a machine that runs continuously, where the figures multiply by twelve. Whether that beats a cloud invoice is worked out in reading your cloud AI bill before you buy a GPU, and the short version is that against the cheapest cloud tier, the electricity alone can cost more than the API.

Heat and noise are not on the spec sheet

Hard to quantify, and decisive for whether you actually enjoy owning the thing.

Power consumption becomes heat. A 575 W card puts 575 W of heat into the room. Run that for hours in a closed room in summer and the room temperature climbs. Cool it and the fans spin, and fans make noise.

Manufacturers rarely publish noise figures and I have not measured any myself, so I will not invent numbers. The structural tendency is enough: more power means more heat to reject, which means higher fan speeds. That relationship is not negotiable.

The remedy is to separate the location. Put the machine in another room rather than under the desk and reach it over the network. Stood up as an OpenAI-compatible endpoint, it does not need to be in the same room — the construction is covered in wiring a local LLM into real work.

Separating brings a second benefit: the desk stays quiet while a long job runs. Batch image generation and speech synthesis can occupy tens of minutes to hours, and a fan running beside you throughout costs concentration. Decoupling how heavy the work is from how pleasant the room is matters more than it sounds.

There is a reason not to separate, too. Over a network, moving large files takes time. If your use is generating a lot of images and pulling them back, that transfer stops being negligible. Decide where you work and where the outputs live, and the choice gets easier.

Three configurations by budget

I said I would not quote prices, but the shapes are worth setting out.

First: use what you already have. If there is a gaming GPU in the house, start with it. Even 8 GB runs an 8B-class model. Find out what your work is actually short of before spending. The point is that you get real decision-making information for zero outlay.

Second: add a discrete GPU. Once you know capacity is the gap, choose on capacity. As the table showed, higher in the range does not reliably mean more memory. Fix the capacity you need, then choose within the set that meets it, and you will not overpay for compute.

Third: a unified-memory machine. For capacity a single card cannot reach, or when quiet operation and a small footprint matter. It cannot be expanded afterwards, so specify it for the largest thing you expect to run, not for today.

If it shares a room with a 3D printer

For readers who make things, a word about the room.

A 3D printer produces heat too, more so in an enclosure where raising the chamber temperature is the point. Add GPU exhaust and both cooling problems get harder at once — the printer wants the chamber warm, the GPU wants to be cold, and they are arguing in the same room.

The practical answer is again to separate them physically, which the network access above makes free.

The other easily missed factor is dust. A room where you sand or cut has fine particles in the air. Cooling fans pull them in and pack them into the heatsink. Do not put heat-generating equipment in the same space as dust-generating work — the principle applies equally to a GPU and to a printer.

You can also build your own airflow. Designing and printing a duct or a stand is a direct application of the ventilation thinking in the inside of an enclosure is hotter than you think. Being able to treat your own working environment as a design problem is one of the quieter advantages of owning a printer.

Measure before you buy

1. Measure your actual free capacity, not the number on the box — what is left after other processes. That tells you how far you can get as things stand.
2. Calculate what the model you want needs, from parameter count and bit width. One multiplication.
3. Target the shortfall plus headroom. Weights are not the whole requirement; context growth and working space are on top. Buy an exact fit and it dies the first time you use a longer context.

Do those three and the capacity you should buy is a number, not an opinion. Everything after that is choosing among products that meet it, on power draw and where it will live. GPU selection feels hard mainly because people start comparing products before fixing the order.

One more pre-purchase check saves regret: is the work you want to run actually suited to local? Something used a few times a month will never build up utilisation. Either you have data that cannot leave, or you have work where more attempts is the win. If neither is true, pause before buying — the reverse guide to local versus cloud is the version of that question with the whole table attached.

Summary — capacity, bandwidth, and somewhere to put it

• The first question is whether it loads, not how fast it runs. Capacity comes before compute.
• Start with roughly 1.5x headroom over the model size.
• Higher in the range does not mean more memory: the 16 GB 5060 Ti holds more than the 12 GB 5070.
• Bandwidth sets generation speed, and MoE models read less than their size implies.
• Unified memory buys capacity as one pool but cannot be expanded later.
• At two hours a day, electricity is small — 465 yen a month for a 250 W card at 31 yen per kWh.
• Power becomes heat becomes fan noise. Separating the machine from the desk solves it, at the cost of transfer time.
• Measure free capacity, calculate what you need, add headroom. Then choose a product.

Sources

• NVIDIA — GeForce graphics card comparison (RTX 50 series memory size and type)
• Apple — Mac mini technical specifications (M4 120GB/s, M4 Pro 273GB/s)
• Apple — Mac Studio technical specifications (M4 Max 410 to 546GB/s, M3 Ultra 819GB/s)
• Japan Electric Appliances Fair Trade Council — reference electricity rate of 31 yen per kWh (Japanese)

ブラウザだけでできる本格的なAI画像生成【ConoHa AI Canvas】

swiftwand

X

AIを使って、毎日の生活をもっと快適にするアイデアや将来像を発信しています。 初心者にもわかりやすく、すぐに取り入れられる実践的な情報をお届けします。 Sharing ideas and visions for a better daily life with AI. Practical tips that anyone can start using right away.