What an AI copilot actually costs to run in production — and how to keep it reliable

The running cost of an AI copilot comes down to four things: how many tokens each request consumes, how quickly it has to respond, how closely you watch it, and how many guardrails you wrap around it. Model fees are usually the smaller share; the engineering that keeps the copilot accurate, fast and safe is the larger one. This piece explains each driver in plain terms and sets out what reliable operation looks like once real users arrive.

Cost driver one: tokens

Every model provider meters usage in tokens, which are fragments of words. A request spends tokens on the way in (your instructions, any retrieved documents, the conversation so far and the user's message) and on the way out (the reply). Output tokens are typically metered at a higher rate than input tokens, and a long, rambling answer costs more than a short, precise one.

The surprise for most teams is how much of the bill has nothing to do with the user's question. A system prompt that runs to several pages is resent on every call. A retrieval step that stuffs ten documents into context when two would do multiplies the input. A chat history that is never trimmed grows with every turn until each message is carrying the whole conversation on its back.

The levers that matter most:

• Prompt discipline. Shorter, tighter instructions, with examples only where they measurably help.
• Prompt caching. Most providers can cache the stable part of a prompt so repeated calls pay a fraction for it.
• Retrieval tuning. Return fewer, better-ranked passages rather than everything that matches.
• Model routing. Send simple classification or extraction to a smaller model and reserve the large one for reasoning.
• Output limits. Cap reply length and ask for structured output where a paragraph is not needed.

Cost driver two: latency

Latency is the cost you feel rather than pay. Two numbers matter: time to first token (how long before the user sees anything) and total generation time. Streaming the reply hides much of the second number; nothing hides the first. If a copilot makes three tool calls and a retrieval query before it starts writing, the user is staring at a spinner for the sum of all of them.

Latency and tokens are linked. Retries on timeouts double the tokens spent on a single request. Running tool calls in parallel, setting sensible timeouts and choosing a faster model for the first response all reduce both. Decide early what your users will tolerate for each type of task; a code suggestion has to appear in under a second, while a summary of a forty-page contract can take a moment.

Cost driver three: monitoring

A copilot that is not observed will drift. Prompts change, models get updated by the provider, your documents change, and quality erodes without anyone noticing until a customer complains. Monitoring is therefore part of the running cost, not an optional extra.

The minimum we put in place:

• Tracing. Every request logged end to end, including retrieval results and tool calls, with personal data redacted before storage.
• Dashboards. Tokens per request, cost per conversation, p95 latency, error rate, refusal rate and user feedback, broken down by feature and by tenant.
• Evaluations. A golden set of a few hundred real questions with graded answers, run automatically whenever a prompt, model or retrieval index changes. Treat it as the copilot's regression test suite.
• Alerts. Budget thresholds per day and per tenant, so a runaway loop is caught in minutes rather than on the invoice.

Cost driver four: guardrails

Guardrails are the controls that stop the copilot doing something wrong, expensive or embarrassing. They are the part most budgets forget, and each one has a price in tokens or latency because many of them are themselves model calls.

• Input checks. Detecting prompt injection in pasted text or retrieved documents before it reaches the model.
• Output validation. Enforcing a schema for structured replies and rejecting anything that does not parse, rather than hoping.
• Scoped permissions. An agent that can read a calendar should not be able to delete one. Give each tool the narrowest access that does the job.
• Human approval. Anything that sends money, emails a customer or changes a record gets a confirmation step.
• Rate limits. Per user and per organisation, so one heavy user cannot consume the shared budget.

Keeping it reliable after launch

Reliability is mostly about assuming things will fail and deciding in advance what happens when they do.

• Abstract the provider. Put the model behind an interface so a deprecation or outage means a configuration change, not a rewrite.
• Fallbacks. If the primary model times out, fall back to a smaller one, or to a cached answer, or to a polite "try again" rather than a blank screen.
• Version prompts like code. Prompts live in the repository, go through review and roll back the same way.
• Pin model versions. Move to a new model version deliberately, after the evaluation set passes, not because the provider changed a default.
• Data handling. Know which region processes your data and what is retained. Associations, government bodies and regulated industries will ask, and the answer must be documented.

An anonymised example

A national franchise network asked us to build a support copilot that answers franchisee questions from an operations manual. The first working version answered well but sent an entire manual section with every question. Trimming retrieval to the three best passages, caching the system prompt and routing simple lookups to a smaller model cut tokens per resolved question by 58% and brought p95 response time under three seconds, with no measurable drop on the evaluation set. The lesson was not that the model was expensive; it was that the first architecture was wasteful.

How we build copilots at OlDevs

OlDevs is a full-stack technology studio in Vancouver, British Columbia. We have been building software since 2014, and our AI development work covers custom machine learning, generative AI and large language models, natural language processing, computer vision, agents and predictive analytics. Every engagement follows the same path: a strategy call, design, a build phase with weekly working demos, launch, then a grow phase where the monitoring described above earns its keep.

You own all of it: the code, the prompts, the evaluation set, the provider accounts and the intellectual property. Copilot interfaces are built to WCAG 2.2 AA and can ship bilingual in English and French. We do not publish prices because no two copilots have the same token profile, latency budget or risk surface. Tell us what you want the copilot to do and we will reply with a quote within one business day.