A running cost model is built from tokens per completed task and tasks per user per day, rather than from a provider headline rate.
Per-request pricing is the wrong unit
The first cost estimate for an AI feature is usually a multiplication: price per million tokens, times the tokens in a typical prompt and reply, times the requests you expect. It is easy to underestimate, because the request in the price list and the request in your product are not the same thing.
One user action rarely maps to one model call. A retrieval feature may rewrite the query, embed it, search a vector index, rerank the candidates, generate an answer, then call the model again to check that answer against its passages. An agentic feature loops: a tool is called, the result is appended, and the model is called again, several times before the task is done.
Retries bill as well. A request that fails before the model starts generating, such as a rate limit or a validation error, is normally not charged; the retry that follows is. A stream the user abandons still bills for the tokens already generated. The unit worth pricing is the completed user task, counting every call the system made to finish it.
What the running cost is made of
Model input tokens cover the system prompt, tool schemas, conversation history and retrieved passages. On a retrieval feature the retrieved passages are often the largest and most variable part, and retrieval decisions set it: how many passages you send, how large the chunks are, and whether you send a whole document where a section would do. Output tokens are priced higher than input on most providers, so response length is worth controlling directly.
Reasoning depth is a separate dial and frequently the largest single per-request lever. Models that reason before answering bill those tokens, and the setting that controls how much they reason moves cost more than most prompt edits. It is worth tuning per route rather than globally: the depth a coding or planning task repays is rarely the depth a classification route needs.
Embedding the corpus once is calculable from document count and size. The recurring parts are easier to miss: every query is embedded too, new and edited documents are embedded continuously, and any change to chunking or the embedding model means re-embedding the whole corpus.
Reranker and vector database pricing varies by product rather than following one scheme. Rerankers are billed per search request with an included document allowance, per block of documents, or per token depending on the vendor; vector databases bill provisioned compute by the hour, read and write units, or stored vectors and queries. Read the specific pricing page rather than assuming a unit, because the unit decides whether widening the candidate set to improve recall is cheap or expensive.
Hosting adds servers, job queues, object storage for documents and audio, egress, and the ingestion workers that run embedding. Tracing is its own line, because useful observability stores full prompts and completions. The last component is the engineering cost of evaluation: a set that has to be maintained as the product changes.
Usage patterns decide the figure
Two numbers do most of the work: tokens per completed task, and completed tasks per active user per day. The second is the one most often left unmeasured, and it moves the total more than the per-token price does. A feature each user touches weekly and a feature in the main path of the product differ by orders of magnitude at the same per-task cost.
Averages hide the shape of the traffic. Session lengths in an assistant are typically skewed: many short sessions and a long tail of much longer ones. Where history is resent on every turn and not cached, a late turn costs well above the first, so a small share of sessions can take a large share of spend. Whether that holds depends on caching, which the next section covers.
Bulk processing needs separate treatment, and is distinct from a provider's asynchronous batch endpoint, which typically bills at around half the standard rate for work that can tolerate delay. Ingesting a corpus at launch is a one-off spike; a pipeline that reprocesses a document on every edit is recurring, driven by the edit rate of the source system rather than by user traffic.
Caching and routing change the shape of the cost
Prompt caching changes which parts of a prompt are expensive. Providers that cache a prefix charge less to read the cached portion and more to write it, so a prefix that is rewritten more often than it is reused can cost more than no caching at all. Longer cache lifetimes are usually available at a higher write price, which pays off only when the gap between requests exceeds the default window.
What can be cached is decided by ordering, because the match is on an exact prefix and any byte change invalidates everything after it. Requests are assembled as tool definitions, then the system prompt, then the messages, so tool schemas sit ahead of instructions in the cached region whether or not that was intended. Stable content belongs first and byte-identical between calls; anything volatile, such as a timestamp or a per-request identifier, belongs after the last cache breakpoint, where it will not invalidate the rest.
Response caching depends on repetition rather than structure. Traffic shaped like a FAQ, where many users ask the same small set of questions, caches well; traffic carrying personal context on every request does not. A response cache also needs invalidating when the documents behind it change.
Model routing sends straightforward tasks to a smaller model and difficult ones to a larger one, turning one rate into a blend. Two effects decide whether it pays. The routing step is itself a call, so the saving has to exceed it, and the comparison depends on what the unrouted default would have been. More importantly, prompt caches are scoped per model, so splitting traffic across two models splits the cache: each model warms its own copy of the shared prefix, and a prefix that was being read cheaply is now written twice. Before building a cascade, measure the simpler option of one capable model at a lower reasoning depth.
What to measure before committing to a build
Instrumenting a prototype is worth more than a careful estimate. Log, per completed task: model calls, input and output tokens for each, cache-write and cache-read tokens separately from uncached input tokens, embedding calls, retrieval and rerank calls with their candidate counts, and call minutes where voice is involved. Hit rates are not billable quantities, so record the token counts themselves; those are what reconstruct the bill. Attach a task identifier so the calls belonging to one user action can be summed.
Report the mean and the distribution. The monthly total is built from the mean cost per task times the task count, so the mean is the number that multiplies into a budget. The median and the high percentiles tell you something different: what a task usually costs, and where the top of the range sits, which is also where context limits and timeouts appear. Collect the corpus facts separately: document count, total size, growth rate and edit rate.
Take the volume assumptions from whoever owns the product: active users, tasks per user per day, peak concurrency, and whether the feature is on by default. Then set a ceiling. A cost per task and per month agreed before implementation makes cost an acceptance criterion alongside the functional ones, and a per-user quota, a cap on retrieved tokens, a bound on tool-loop iterations and a reasoning-depth setting are the controls that hold it.
Building the model from expected volume
A useful cost model is a calculation over volume rather than a single figure. It states the cost of one completed task as a sum of its components, states the assumed volume, and multiplies. Written that way it can be checked line by line, and recomputed when provider prices change or when real usage differs from the assumption.
Every AIVETech engagement starts with a fixed-price discovery sprint at USD 2,500 over two weeks, and a running-cost model is one of its deliverables, alongside the architecture, the acceptance criteria and a fixed price for the build. The sprint fee is deducted from the build. The model is built from the provider choices and retrieval design settled during the sprint, and from your expected volume, so it is an estimate to check against measurement rather than a figure to rely on unexamined.
Read about the AI Feature Implementation Sprint or discuss an AI project.