How do you control LLM costs?
You control LLM costs by measuring every request, allocating its cost to a team and a model, and then applying levers where the data shows waste: approved model lists, budgets with caps, caching and shorter prompts, routing, refusing requests before they reach the provider, and renegotiation. Report realised savings separately from modelled ones, because only realised savings change the invoice.
The FinOps Foundation's guidance on FinOps for AI starts from the same point: price multiplied by quantity still gives cost, so you manage either the rate or the usage. What changes with AI is the unit, the token, and how hard it is to know who consumed it.
Why AI spend is hard to attribute
- Shared keys. One API key used by several applications produces one total on the invoice, with no team or purpose behind it.
- Many teams. Product, engineering, data and business teams each start their own experiments, often on a company card.
- Several providers. Model providers, cloud marketplaces and self-hosted models each bill, or cost, in a different way.
- AI inside SaaS. AI features bundled into existing subscriptions appear as a higher licence fee, not as usage.
Allocation needs either a control point that knows who made each request, or keys issued per team with usage imported from each provider's administration interface. In practice you need both: the first for routed traffic, the second for everything else.
What to measure
| Metric | Why it matters |
|---|---|
| Input and output tokens per request | The billing unit. Providers often price input and output tokens differently, so the split matters |
| Model requested versus model served | Shows whether routing or fallback changed the model, and therefore the price |
| Actual cost per request | The figure finance needs, calculated from the model served and its price |
| Cost by team and by model | The basis for showback, chargeback and budget conversations |
| Refused requests | Requests stopped by a policy or a budget, which cost nothing but show demand |
Levers for reducing LLM cost
Each lever needs something in place before it works, and each has a cost of its own. The table does not estimate savings, because they depend entirely on your current usage.
| Lever | What it needs | Trade-offs |
|---|---|---|
| Approved model lists and right-sizing | A list of models per use case, and a test showing a smaller model is good enough | Quality may drop for some tasks. Lists need maintaining as models change |
| Budgets with hard and soft caps | Spend attributed per team, and an agreed budget for each | Hard caps can interrupt a service. Soft caps only alert |
| Caching and prompt size | Visibility of repeated prompts and long system prompts or context | Engineering effort. Stale cached answers if not designed carefully |
| Routing | A control point between applications and providers, and rules for which requests go where | Another component to operate. Behaviour can differ between models |
| Refusing requests before the call | Policy checks at admission: identity, model, data, budget | A refused request is a user who did not get an answer, so refusals need clear reasons |
| Renegotiation | Reliable volume data by provider and model over several months | Commitments reduce flexibility if usage or preferred models change |
Realised versus modelled savings
A cost programme produces two kinds of number. Realised savings are reductions you can see in actual costs: a lower invoice, or a lower cost per request after moving a workload to a smaller model. Modelled savings are estimates of avoided cost: the price a refused request would have had, or what a team would have spent without a cap.
Both are useful. Modelled savings show where controls are active and what demand looks like. But they rest on assumptions, such as that the refused request would otherwise have been made in the same form. Finance should keep them in separate lines, and never add them together, so that a reported saving can always be traced to the invoice or clearly marked as an estimate.
A single savings figure that mixes realised and modelled amounts will not survive a budget review. Two lines, with the method for each written down, will.
A four-step programme
- Make spend visible. Import usage from each provider's administration interface and route application traffic through one control point, so that every request has a team, a model and a cost.
- Allocate and agree budgets. Show each team its spend by model, then set budgets with them. Start with soft caps while the figures settle.
- Apply levers where the data points. Right-size the largest workloads, set approved model lists, reduce prompt size and caching where repetition is high, and move confident budgets to hard caps.
- Review and renegotiate. Report realised and modelled savings separately each month, and use several months of volume data when you next negotiate with providers.
LLM cost control checklist
- An inventory of AI providers, keys and SaaS AI features, each with an owner
- Usage imported from every provider that offers an administration interface
- A record per request of tokens, model requested, model served and actual cost
- Spend reported by team and by model
- An approved model list for each main use case
- A budget per team, with a decision on hard or soft caps
- Refused requests logged with their reason
- Realised and modelled savings reported on separate lines
- A provider review date backed by volume data
How MERIDVAR helps
MERIDVAR is self-hosted software you install and operate on your own network. Nothing is sent to MERIDVAR.
- Cost ledger. Every admitted request is recorded with its tokens, model requested, model served and actual cost, and spend is reported by team and by model.
- Budgets at admission. Team budgets with hard or soft caps are checked in the admission gateway, before the request is sent.
- Usage import. Usage can be imported from Anthropic and OpenAI administration keys, for traffic that does not pass the gateway.
- Approved model lists. Requests for models outside the list are refused at the gate.
- Nothing billed on refusal. A refused request never reaches the model provider, so nothing is billed for it.
MERIDVAR does not promise a level of savings. Controls apply only to traffic routed through the gateway, and the product is in beta. See AI FinOps for details.