Decision support

Do we need our own LLM, or is an API enough?

Almost always an API. Three situations genuinely justify self-hosting, and “security” is usually not one of them.

The question

Should we run our own model instead of calling someone’s API?

The short answer

Default to an API. Self-hosting replaces a variable cost you can turn off with a fixed cost you cannot, and it moves model operations onto your team — quantisation, batching, KV-cache management, capacity planning, upgrades. Three cases genuinely justify it: a legal data-residency or air-gap requirement you cannot satisfy contractually; steady, high, predictable volume where a reserved GPU beats per-token pricing; or a workload where a small fine-tuned model on cheap hardware outperforms a frontier model on a narrow task. Note what is not on that list — “our data is sensitive” is usually a contract problem, not a hosting problem, and the frontier providers all offer zero-retention endpoints. The honest test is a spreadsheet, and most teams have not built it.

Written for

Leaders being asked to fund GPU capacity
Teams told by security that data cannot leave the building
Anyone comparing an open-weight model to a frontier API

Why the answer is that

The argument for self-hosting almost always arrives as a security argument and almost always turns out to be an economics argument that nobody has run. Security first, because it is quickest to settle: if your objection is that a third party might train on your data, that is contractual and already addressed by zero-retention endpoints. If your objection is that data must not leave a jurisdiction, check whether your provider has a region there — several do — because that is far cheaper than operating your own inference. Only a genuine air-gap requirement, or a regulator that names self-hosting specifically, actually forces the decision.

Then the economics, which is where it gets interesting and where the answer is genuinely sometimes yes. Per-token pricing is a variable cost: at low volume it is close to free, and it scales linearly. A GPU is a fixed cost: you pay for it idle. So there is a crossover, and it depends almost entirely on utilisation. A reserved instance running at 10% utilisation is dramatically worse than an API. The same instance at 70% utilisation, on a workload that never spikes, can be several times cheaper. The question is not “is self-hosting cheaper” but “what is our sustained utilisation, and how confident are we in that number.”

What the spreadsheet usually leaves out is people. Self-hosting means someone on your team now owns inference performance: choosing a quantisation that does not degrade quality in ways your evals do not catch, tuning batch size against latency targets, managing KV-cache memory, planning capacity for launches, and repeating some of that on every model upgrade. That is a meaningful fraction of an engineer indefinitely. At Indian or European salary levels this may still work out; the point is it belongs in the comparison and it usually is not there.

There is a third case that is legitimately compelling and gets discussed least: a small model fine-tuned on one narrow task can beat a frontier model at that task, on hardware that costs very little. Classification, extraction, routing, intent detection. If a large part of your token spend is a repetitive, well-specified task, moving just that task to a small self-hosted model — while keeping the frontier API for everything hard — is usually a better move than migrating wholesale. It is also the only version of this decision that is easy to reverse.

Reason given versus whether self-hosting actually solves it

Work down the list. If your reason is in the top half, self-hosting is probably not the answer.

Reason for self-hosting	Does it hold up?	Cheaper alternative
Our data is confidential	Rarely — this is a contract question	Zero-retention endpoint plus a signed DPA
The provider will train on our data	No — explicitly excluded on business tiers	Read the terms once, confirm in writing
Data must stay in-region	Sometimes — depends on your provider’s regions	A provider region in the same jurisdiction
We need to avoid vendor lock-in	Partly — but you gain hardware lock-in	An abstraction layer plus evals that run against two providers
We want lower latency	Sometimes — you remove a network hop, add queueing	Prompt caching, smaller model for easy turns, streaming
API costs are too high	Only if utilisation is high and steady	Model routing and caching first — usually a bigger win, no capex
Genuine air-gap or named regulatory requirement	Yes — this actually forces it	None. Budget for it properly
High, steady, predictable volume	Yes — reserved GPU beats per-token at high utilisation	None, but measure utilisation before committing
One narrow repetitive task dominates spend	Yes, and this is the best version	Self-host just that task; keep the API for the hard ones

The top six rows cover most of the conversations I have. If your reason is in that half, do model routing and caching first — those cut spend without a capital commitment and without adding inference to your team’s job description.

What belongs in the comparison

Both columns, or the comparison is not honest. The lines teams forget are marked.

Cost line	API	Self-hosted
Per-request compute	Per-token, scales to zero	Included in the instance you already paid for
Idle capacity	None — you pay for what you use	Paid for continuously — the decisive variable
Traffic spikes	Absorbed by the provider	You over-provision, or you queue
Model upgrades	Change a version string	Re-evaluate, re-quantise, re-tune, re-deploy
Inference engineering time — often omitted	None	A material fraction of an engineer, indefinitely
Eval suite	Needed either way	Needed either way, and needed more
Quality risk from quantisation — often omitted	None	Real, and invisible without evals
Exit cost	Change a version string	Hardware commitment and migration

If the two columns are close, choose the API — it is the reversible option. Self-hosting should win clearly on the numbers or not at all, because you cannot cheaply undo it once the capacity is bought and the expertise is built.

Get the answer for your own system in four minutes

The free scorecard produces a score out of 100, a breakdown across reliability, cost and observability, and your gaps ranked by production risk with a fix and effort estimate for each. It prints to a PDF you can take into a planning meeting — which is usually more persuasive than an argument.

Run the scorecard

If you are the engineer who found this page: Written to be forwarded. If you are the engineer who found this, send the relevant one up — it is framed for the person who has to approve the work.

If you decide to do the work

The engagements this decision leads to, each with the full method and effort estimates — whether your team runs it or I do.

LLM cost reduction

A materially lower bill, with evidence that quality held.

The method

Agent eval suites

A suite your team owns that fails the build when agent quality drops.

The method

Questions

Do we need our own LLM to keep our data private?

Usually not. Data privacy with a hosted model is a contractual matter: business tiers exclude training on your data, and zero-retention endpoints mean requests are not stored.

Self-hosting is genuinely required when you have an air-gap requirement, or a regulator that names self-hosting specifically, or a data-residency obligation your provider has no region for. Those are real but far less common than the security conversation implies.

When is self-hosting an LLM actually cheaper than an API?

When utilisation is high and predictable. A GPU is a fixed cost you pay while idle, so the comparison turns almost entirely on sustained utilisation.

At low or spiky utilisation an API is dramatically cheaper. At high steady utilisation a reserved instance can be several times cheaper. Before committing, measure your actual sustained request rate — and do model routing and caching first, because those usually cut more cost than self-hosting and require no capital commitment.

What GPU do we need to run an LLM for our agent?

It depends on model size, quantisation and how much concurrency you need, and the honest answer is that if you are asking this question first you are solving the problem in the wrong order.

Decide the workload before the hardware. A narrow classification or extraction task can run on a small model and modest hardware. General reasoning at frontier quality needs enough memory for the weights plus KV cache for every concurrent request, which is where teams underestimate — concurrency, not the weights, is usually what forces a bigger card.

Is there a middle option between self-hosting everything and using an API?

Yes, and it is usually the best answer. Move one narrow, high-volume, well-specified task — classification, extraction, routing, intent detection — to a small self-hosted model, and keep a frontier API for everything that needs real reasoning.

That captures most of the cost saving, needs far less hardware, and is easy to reverse if the quality is not there. Wholesale migration is the version that gets regretted.

Want a second opinion on the decision?

Describe your situation and I will tell you what I would do — including when that is "nothing" or "handle it internally". No charge for that, and it is genuinely how a lot of these conversations end.

Message me Book a 20-minute call or email instead

Other decisions