The first significant AI invoice usually arrives as a surprise. Not because anyone was careless, but because nobody could have predicted it. The pilot cost a rounding error. Production costs a line item that finance now wants explained, and the team explaining it often cannot say which decisions produced the number.
AI spend behaves differently from most infrastructure cost. It scales with usage in ways that are hard to intuit, and it is determined largely by decisions taken months earlier at design time. By the point the invoice is legible, the expensive choices are already deployed.
The expensive choices are made at design time, long before the invoice is legible.
Cost is a design-time decision
Consider a retrieval-based assistant. How much context is sent with each request. How aggressively results are cached. Whether every query goes to the largest available model or is routed by difficulty. How often the knowledge base is re-indexed. Where the data physically sits relative to the compute. Each is an architectural decision, and together they can produce an order of magnitude difference in monthly cost for a system that behaves almost identically to the person using it.
- Model routingSend only the hard requests to the largest model
- Context sizeRetrieve what the task needs, not everything available
- CachingReuse answers to repeated and near-identical questions
- Data localityKeep data and compute in the same place
Each is decided during design. Retrofitting any of them usually means rebuilding the request path.
The decisions that actually move the number
Three tend to dominate. The first is model routing: sending every request to the most capable model is the simplest design and usually the most expensive, when a large share of traffic would be served just as well by something smaller. The second is context discipline, because retrieving and sending more than the task requires is invisible in testing and costly at volume. The third is caching, which is frequently omitted altogether since it was unnecessary at pilot scale.
None of these are optimizations to apply later. They are structural, and retrofitting them usually means rebuilding the request path.
Make cost observable before you make it optimal
Teams often jump to reducing cost before they are able to see it. The more useful first step is attribution: cost per feature, per tenant, per request type. Until spend can be traced to something a business person recognizes, every conversation about it is speculative, and the usual outcome is a blunt decision to limit usage.
Attribution also changes the conversation from cost to value. A feature costing a meaningful amount each month is a problem in isolation and a bargain if it replaces hours of manual work. Neither judgement is available without the numbers.
Governing spend without blocking work
Budgets and alerts belong in the architecture rather than in a monthly review. Set an expected cost envelope per use case at design time, instrument against it, and alert when reality diverges. Treat a sustained overrun the way you would treat a performance regression: something to investigate, not something to absorb quietly.
AI economics are not inherently unpredictable. They are predictable for organizations that treat them as an architectural property rather than an operational surprise, and that decide the cost profile at the same time they decide everything else about the system.
Key takeaways
- AI cost is set at design time by model routing, context size, caching and data locality.
- Sending every request to the largest model is the simplest design and usually the most expensive.
- Attribute spend to features and tenants before attempting to reduce it.
- Set a cost envelope per use case and alert on divergence, as you would for a performance regression.