How to budget LLM running costs before you write any code

You can estimate what an LLM feature will cost to run before anyone writes a line of code, and you should. The method is simple: count the tokens a typical request sends and receives, multiply by the published per-token rates for your chosen model, then scale by realistic request volumes. The answer will not be exact, but it will land in the right order of magnitude, which is what a budget decision actually needs.

What actually drives the bill: tokens in, tokens out

Every request has two meters running. Input tokens cover everything you send: the system prompt, retrieved documents, conversation history, tool definitions and the user's message. Output tokens cover everything the model generates. On published pricing across the major providers, output tokens carry a higher rate than input tokens, so the shape of your feature matters as much as its volume.

A few rules of thumb keep the estimate honest. Tokens are not words: a token is usually a fragment of a word, so count a representative prompt with your provider's tokenizer rather than guessing from a character count. Code and JSON are denser in tokens than prose of the same visual length. Conversation history compounds: in a naive chat implementation, every prior turn is resent as input on every new turn, which means input cost per turn grows with conversation length. Tool definitions and structured-output schemas are input tokens too, and agentic features that loop through several model calls per user action multiply everything by the number of steps.

Write these down per feature, per request type. "Summarise a document" has a very different token profile from "answer a support question with retrieval" or "run an agent that calls four tools." Lumping them together is the first way estimates go wrong.

The envelope-math method, step by step

Here is the whole method. It fits on one page and a product manager can check it.

1. Define request types. List each distinct way your feature calls a model: chat turn, summarisation, classification, embedding, agent step.
2. Estimate tokens per request. For each type, write down expected input tokens (prompt plus context plus history) and output tokens (typical response length, plus a cap you will enforce).
3. Price one request. Cost per request is input tokens divided by one million times the input rate, plus output tokens divided by one million times the output rate, using the provider's published per-million-token rates, current at the time of writing.
4. Model the volume. Requests per month is active users times sessions per user times requests per session. Estimate low, expected and high cases rather than a single number.
5. Add multipliers. Retries, agent loops, evaluation runs, and internal testing all consume tokens. A simple overhead multiplier on top of user-facing volume keeps them from surprising you.

Multiply it through and you have a monthly envelope: pessimistic, expected and optimistic. If the pessimistic case is affordable, proceed. If even the optimistic case is not, redesign now, while redesign is cheap.

How does caching change the math?

Prompt caching is the single most overlooked lever in early estimates. The major providers offer cached-input pricing, current at the time of writing: when the opening portion of your prompt is identical across requests, those repeated tokens are billed at a discounted rate. System prompts, tool definitions, few-shot examples and long reference documents are all prime candidates, because they are identical on every call.

The adjustment to the envelope is one line. Split input tokens into a stable prefix and a variable suffix, then work out the effective input cost as the prefix priced at the cached rate for the share of calls that hit the cache, plus the prefix priced at the standard rate for the share that miss, plus the suffix at the standard rate. For a feature with a long system prompt and short user messages, the stable prefix can account for the bulk of your input tokens, and the discount flows straight through to the bottom line.

Caching also changes design decisions. It rewards putting stable content first and volatile content last, keeping prompts deterministic, and batching related requests close together in time so cache entries stay warm. These are architectural choices, which is exactly why the cost conversation belongs before the build, not after.

When does model routing pay off?

No rule says every request deserves your most capable model. Routing sends each request to the cheapest model that can do that job well: a small fast model for classification and extraction, a mid-tier model for everyday drafting, the frontier model only for the hard cases. Blended cost is then the volume-weighted average, the sum across models of the share of traffic each one handles times its cost per request.

Because smaller models are priced well below frontier models, moving routine traffic down a tier reshapes the budget. The table below compares the main cost levers side by side.

Routing has a quality cost you must verify, not assume. Build a small evaluation set for each request type and confirm the cheaper model clears the bar before you route traffic to it. Our AI development practice treats routing and evaluation as one piece of work for exactly this reason.

A budget built on tokens you counted beats a forecast built on hope. Estimate on paper, verify in logs, and let the two argue until they agree.

Modelling load: budget for the worst realistic day

Averages hide the day that hurts. Usage of interactive features is spiky: launch announcements, seasonal peaks, one enthusiastic team adopting the tool, a workflow that embeds your feature in a loop. Take your expected daily volume, ask what the worst realistic day looks like, well above the average, and confirm that day is affordable and, just as important, servable.

Servable is where rate limits enter. Providers cap requests per minute and tokens per minute by usage tier. Convert your peak load into those units: peak requests per minute times tokens per request must fit inside your tier's tokens-per-minute ceiling, with headroom for retries. If it does not, you need a higher tier, provisioned capacity, a second provider, or a queue that smooths the spike. Each of those is a design decision, and each is far easier to make before the architecture exists. Where the LLM sits inside a wider system, capacity planning belongs alongside the rest of your cloud infrastructure decisions rather than in a separate spreadsheet.

Finally, put guardrails in the product itself: per-user and per-organisation usage limits, sensible defaults for context length, and an alert when spend deviates from the envelope. A budget without an alarm is a hope.

Contracts, commitments and keeping your options open

Once the envelope exists, it becomes a negotiating tool. Providers offer committed-use arrangements, batch discounts and enterprise agreements; your envelope tells you whether a commitment is safe to sign, because you know your floor as well as your ceiling. Commit near the floor, stay flexible above it.

Protect your optionality in the code as well as the contract. Keep prompts and model choices in configuration rather than scattered through the codebase, so a price change or a better model becomes a config change instead of a rewrite. Log input tokens, output tokens, cache hits and model used for every request from the first deployment. That log is what turns your envelope into a real budget: each month, reconcile actual token consumption against the estimate, find the request types that drifted, and adjust. Model pricing has changed often since these APIs arrived, and capable cheaper models keep appearing, so a team that can re-route quickly keeps capturing those improvements.

How OlDevs helps

OlDevs is a full-stack technology studio in Vancouver, building software since 2014. Our AI work covers generative AI and LLM applications, AI automation and agents, and the evaluation discipline that keeps them honest. Cost modelling is built into how we scope: before we build an LLM feature, we put the envelope math in front of you, with the caching, routing and load assumptions written down where your team can challenge them. You see a working demo every week, you own all code, designs, accounts and IP, and one accountable team carries the project from estimate to production. If you would like to see how that process runs in practice, read about how we work.

Planning an LLM feature and want a grounded number before you commit? Request a quote and we will reply within one business day.