Deciding What Runs Where — A Reverse Guide to Local vs Cloud AI

There is one reliable trap in running AI on your own hardware: trying to move everything. You move it, the speed is not there, the quality is not there, you go back to the cloud, and the only thing left over is the conclusion that local did not work.

The realistic answer is not a binary. It is an allocation problem — deciding where each job lives. This article puts the routing criteria in a form you can look up by task, and sets out three starting configurations. Every figure is from an official source; assumptions are labelled as assumptions.

Stop treating it as a binary

What makes the decision hard is looking at the cloud as one thing. Inside it, prices vary enormously.

Per million tokens, taken from each vendor official page on 19 August 2026. On output, top to bottom is more than twenty times.

So the choice is not cloud or local. It is three layers: top-tier cloud, cheap cloud, local. Start thinking in three and the decision gets much easier.

Two moving parts to note. The $2 / $10 on Claude Sonnet 5 was announced as introductory pricing through 31 August 2026; the official page now states the increase scheduled for 1 September will not take place and this is the standard price. Gemini 3.7 Flash holds $0.75 / $3.75 through 31 December 2026 and doubles on 1 January 2027. Rates move, so check them each time you decide.

A routing table you can look up

Two rows for code generation because the requirements differ. Two rows for images for the same reason: producing ten and keeping one is overwhelmingly better locally, while a single image that has to be right removes the disadvantage of metered pricing.

Fix the order of the questions

More reliable than consulting the table is having an order to work through.

1. Does this job have a data-egress constraint? If yes, local is the only answer and you stop here.
2. If not, try a cheaper cloud tier first. The number of jobs Luna or Flash handle fine is surprising.
3. If volume is stacking up the bill, make it a candidate for moving.
4. Before moving it, verify on one small job that a local model clears your quality bar.

Most people skip step two and land on step four. The table above shows the cost of that: dropping a tier inside the cloud already divides the bill by more than twenty. Worth trying before buying hardware.

A tip for step two: do not drop everything at once — compare on one job. Take a process you are actually paying for, run the same input through the top and the cheap tier, and look at both outputs. It takes minutes and costs almost nothing.

Judge on one question, not overall quality: does it clear the bar this job needs? Accuracy for classification, structural integrity for format conversion, whether the thing you cannot afford to lose survived for a summary. If it clears, settle on the cheap tier and stop deciding.

Skip that check and you keep paying more than ten times the rate for unfounded reassurance. The fact that you are using a top model is not evidence that you need one.

Work out the break-even

If cost is the reason for moving, get the crossing point. Initial outlay divided by monthly saving equals months to recover.

Assumptions: thirty calls a day at 8,000 input and 2,000 output tokens, so 7.2M and 1.8M tokens over thirty days. Exchange rate assumed at 150 yen to the dollar. Local side assumed at 300 W for two hours a day, which at the reference rate of 31 yen per kWh is 558 yen a month. GPU outlay assumed at 150,000 yen. The reference rate comes from the Japan Electric Appliances Fair Trade Council, which notes as the council itself states that actual charges depend on your retailer.

Against the cheapest tier, local is more expensive — 558 yen of electricity against 540 yen of API. If cost is your only reason, the arithmetic says stay. The full working is in reading your cloud AI bill before you buy a GPU.

Work out the capacity you need

Once a job is going local, the next question is whether it fits. Bytes required = parameter count x bits per parameter / 8. Llama-3.1-8B at Q4_K_M is published at 4.8944 bits and 4.58 GiB, and the formula reproduces it.

Two things to remember. Q4 does not mean four bits — the real figure is about twenty percent higher. And for a mixture-of-experts model, memory follows total parameters, not active ones. Qwen3.6-35B-A3B activates 3B and still needs all 35B resident. Full treatment in estimating what fits in your VRAM.

Three configurations by budget

Use what you already own. A gaming GPU with 8 GB runs an 8B-class model. Zero outlay, and you learn what your work is short of.

Add a discrete GPU. Choose on capacity, not on position in the range — the 16 GB RTX 5060 Ti holds more than the 12 GB RTX 5070.

Buy a unified-memory machine. For capacity beyond a single card, or when quiet matters. Not expandable later, so specify for the largest thing you expect to run. Card-by-card figures are in what to buy first for local AI.

Decide the runtime and the connection

Every major runtime — llama.cpp, Ollama, LM Studio, vLLM — exposes an OpenAI-compatible endpoint, so switching between local and cloud is a base-URL change, provided you built the switch point at the start. Retrofit it and you will not bother, and everything will drift to one side. Tool calling and MCP are covered in wiring a local LLM into real work.

Do not check the licence last

For text generation the licence rarely decides anything. For image generation it decides everything. FLUX.1 [dev] confines use of the model to non-commercial purposes while permitting commercial use of its output, and revenue-generating activity is expressly outside the non-commercial definition. FLUX.1 [schnell] is Apache 2.0 and the question disappears. Read the clauses before you build the pipeline, not after — covering your article images locally goes through them.

Design the long jobs first

Batch image generation and speech synthesis both run for tens of minutes to hours, and both fail eventually. Decide how far a failure rolls you back before you start. Writing each image to disk as it completes keeps the finished ones; a pipeline with no per-chunk resume means one interruption costs the whole run. The measured version of that, including 56 chunks where 18 were expected, is in turning long articles into audio with local TTS.

How it splits at a 3D printing bench

Concretely, for people who make things. Anything touching design data stays local — with a customer drawing it is a requirement, not a preference. Ideation and technical research can go to the top cloud tier: low sensitivity, and quality drives the result. Product photo backgrounds and explanatory diagrams are local image generation, because you need volume and the hit rate is low. The hero visual on a sales page may justify the cloud, since one image moves revenue. Print failure detection is local only — streaming camera frames out continuously is not realistic on bandwidth. Reading technical documents aloud while you work is local, because length sits badly with metered pricing.

Laid out like that, local and cloud are barely competing. Each has a region it owns, and the overlap is smaller than you expect.

One production run, using both: specification work and research on the top cloud tier; sweeping shape parameters locally; fixed to local the moment a dimensioned drawing is involved; product copy drafted on a cheap cloud tier; photo backgrounds local; only the hero visual back to the cloud.

Note how often the switch happens inside one process. That is exactly why building it so the endpoint can be swapped pays off. If changing sides means editing configuration each time, the friction wins and everything collapses onto one side.

The commercial half of this — pricing, formats, the paperwork — is in deciding what to sell before you print it. Allocating AI and designing the business make more sense looked at together than separately.

Summary — three layers, decided in order

• Not a binary: top-tier cloud, cheap cloud, local. Going down a tier divides output cost by more than twenty.
• Route on two axes — is failure cheap, and can the data leave.
• Order the questions: data constraint, then cheap tier, then volume, then a quality check.
• Break-even is outlay divided by monthly saving: about 13 months against a top model, never against the cheapest tier.
• Capacity is parameter count x bits per parameter / 8, on total parameters for MoE models.
• Every runtime is OpenAI-compatible, so build the switch point first.
• Licences decide image generation, not text. Read them before building.
• Long jobs need a rollback plan before the first run.

Sources

• Anthropic — Pricing (official Claude API rates)
• OpenAI — API Pricing (GPT-5.6 tiers)
• Google — Gemini API Pricing
• llama.cpp — quantize README (bits per parameter and file size by format)
• Japan Electric Appliances Fair Trade Council — reference electricity rate of 31 yen per kWh (Japanese)

ブラウザだけでできる本格的なAI画像生成【ConoHa AI Canvas】

swiftwand

X

AIを使って、毎日の生活をもっと快適にするアイデアや将来像を発信しています。 初心者にもわかりやすく、すぐに取り入れられる実践的な情報をお届けします。 Sharing ideas and visions for a better daily life with AI. Practical tips that anyone can start using right away.