Everyone budgets for training and gets surprised by inference. Training is capital: you spend it once and amortize it. Inference is an operating cost you pay on every single request, forever, and it scales with success. The more people use the thing you built, the more it costs to run. That’s the opposite of how software economics are supposed to work, and if you don’t understand the cost structure, growth arrives feeling like a tax.
Good news is the bill is legible. Inference cost isn’t a black box, it’s a handful of variables you can name and measure and pull on. Most teams overpay not because the technology is expensive but because nobody ever looked at which lever was stuck.
Tokens are the unit of cost
You pay per token in both directions, and the two directions aren’t priced alike. Input tokens (prompt, context, history) are usually cheaper than output tokens, but there are vastly more of them. A long system prompt repeated on every request is a recurring charge most people forget they signed up for.
That reframes prompt design as cost engineering. Every token in your window is a token you pay for on every call, so the discipline shifts from “what could I include” to “what must I.” The agent that re-reads the whole codebase each turn isn’t thorough. It’s expensive. The one handed exactly the three files it needs does the same work for a tenth of the price.
Cache the prefix or pay for it twice
The single biggest win available to most teams is caching, and it’s routinely left on the floor. If a large chunk of your prompt is identical across requests, a fixed system prompt, a shared instruction block, a document you keep asking questions about, the provider can cache the computed representation of that prefix and reuse it.
The economics are stark. A cache hit on a long shared prefix can cost a fraction of recomputing it, sometimes an order of magnitude less. The catch is that caching is prefix-sensitive. The cacheable part has to come first and stay byte-identical. Reorder your prompt, inject a timestamp near the top, vary the instructions per request, and you blow the cache without ever seeing an error.
# cache-hostile: the variable part is at the front, so nothing caches
prompt = f"User {user_id} at {now()}\n\n{HUGE_SYSTEM_PROMPT}\n{question}"
# cache-friendly: stable prefix first, variable suffix last
prompt = f"{HUGE_SYSTEM_PROMPT}\n\nUser context: {user_id}\n{question}"
# ^^^^^^^^^^^^^^^^^^^^ identical across calls -> cachedBoth prompts ask the same question. One pays full price for the system prompt every time. The other pays once and reads from cache after that. I’ve seen a single misplaced timestamp account for most of a team’s monthly spend, and it took a week to find because nothing about it looks broken.
Batching and the latency tradeoff
Underneath the API, inference servers are throughput machines. They process in batches because a GPU running one request at a time is a GPU mostly sitting idle. Bigger batches mean higher throughput and lower cost per token, and they also mean an individual request may wait a few milliseconds for its batch to fill.
So throughput and latency pull against each other and you have to pick a point on that curve deliberately. A user-facing chat wants low latency and will pay for smaller batches. A nightly bulk job wants maximum throughput and doesn’t care whether any single item waits. Run both through the same path at the same settings and one of them is paying for a tradeoff it never needed.
Teams that keep the bill sane route by workload. Interactive traffic gets latency-optimized serving, batch traffic gets throughput-optimized serving, cacheable prefixes get cached, and context gets trimmed to what the task requires. None of this is exotic. It’s refusing to pay for capacity, tokens, or latency you didn’t need, which is the same discipline that makes any system at scale affordable.