AI Inference Spend: Build an Outcome Ledger Before Scaling Usage

By DataTip · Published September 1, 2026

An AI workload should not scale simply because users are adopting it. The more important question is whether each additional request creates enough value – in revenue, service quality, labour saved, or risk reduced – to justify its inference spend. AI Inference Spend: Build an Outcome Ledger Before Scaling Usage starts with that decision rather than with a cheaper model.

AI inference spend is driven by token volume, request structure, and architecture as much as by nominal model rates. Leaders should track input, output, cached, reasoning, and batch usage by feature or task, then connect that unit cost to delivered value. ProjectDiscovery showed how prompt-memory restructuring and cache controls changed economics without changing models, features, or traffic.

ProjectDiscovery offers a useful example. The company behind the open-source Nuclei security scanner reported that its cache hit rate rose from 7% to 74%, then to 84%, after changing how prompt memory was packaged and adding explicit cache controls. Across 9.8 billion cached tokens, those changes reduced total LLM spend by 59% to 70%. The models, features, and traffic stayed the same. The structure of the requests changed.

That is the practical inference-cost lesson: the nominal token rate matters, but volume, request structure, and architectural defaults often matter more. Before approving more usage, you need an outcome ledger that connects consumption to the work the system actually performs.

What is LLM inference, and why does it create ongoing spend?

LLM inference is the runtime process in which a trained model turns a prompt into a response, token by token, using accelerator hardware. Training creates the model once; inference runs it repeatedly for every request, user, feature, and automated workflow.

That distinction matters for planning. Training is usually treated as a discrete investment. Inference is an operating expense that grows with adoption. More users, more product features, and more agentic workflows can all increase consumption, so a successful AI feature can accumulate a larger inference bill over time than the original model-training event.

The business decision is therefore not simply whether a model is expensive. It is whether the workload’s recurring cost remains proportionate to its delivered outcome. A feature that improves service quality may justify its consumption; a feature that generates activity without a measurable benefit may not.

An outcome ledger should make that distinction visible. At minimum, it should associate each workload with:

• The feature, customer journey, or operational process it supports
• The tokens consumed and the model configuration used
• The relevant outcome, such as revenue, service quality, labour saved, or risk reduced
• The owner responsible for reviewing whether continued usage is justified

How does token-based billing form an inference bill?

Inference billing is built around tokens, which are small pieces of text – roughly three quarters of an English word in typical English usage. Providers generally quote separate rates for input tokens and output tokens, so the bill depends on both what you send to the model and what it generates.

Input includes the prompt, system instructions, conversation history, retrieved documents, and other context. Output is the model’s generated response. Output is typically priced at about five times the input rate, which makes unnecessary verbosity a recurring cost issue rather than a one-off detail.

A useful way to understand the meter is to separate the main billing categories:

• Standard input: The ordinary prompt and context sent with a request.
• Standard output: The response generated for the user or calling system.
• Cached input: Reused prompt content that qualifies for a lower cached-input rate under the provider’s rules.
• Reasoning tokens: Internal model work that may be billed as output even when the user never sees the reasoning.
• Batch processing: Work submitted under batch terms, where supported, for workloads that do not require an immediate response.
  These categories do not affect every workload in the same way. Your effective token cost is a blend weighted by the architecture: how much context is repeated, how long outputs are, whether reasoning is enabled, and whether work can run in a batch.

This is why a model selector alone is a poor cost-control mechanism. Two teams can use the same model and receive very different bills because one sends long retrieved context, triggers multiple calls, or prevents caching while the other does not.

Why do simple, retrieval, and agentic tasks produce different bills?

The number of tasks is not enough to predict inference spend. The calls inside each task, the size of the context, and the amount of generated output determine how much work the model performs.

The source provides three illustrative workloads using a blended workhorse rate. The exact monetary result depends on the provider rates in force, so the important comparison is the consumption pattern:

• Simple chat reply: 500 input tokens and 300 output tokens per task
• RAG query with retrieved context: 6,000 input tokens and 500 output tokens per task
• Agentic task with 12 chained calls: 60,000 input tokens and 15,000 output tokens per task
  At the same task volume, the agentic workflow consumes dramatically more tokens than the simple chat interaction. A retrieval-based answer sits between them because the system sends additional context, even though its visible response may remain relatively short.

This is where an outcome ledger becomes useful. Rather than reviewing aggregate monthly tokens, review spend per feature, customer, or task and compare it with the outcome that justifies the workload. If an agentic task costs more because it completes valuable work, that may be a sound trade-off. If the extra calls only create internal activity, scaling them deserves scrutiny.

Why can total inference spend rise while token rates fall?

Cheaper model access can increase total consumption instead of reducing total spend. This pattern is known as the Jevons paradox: when an input becomes cheaper, people often use more of it, and the increase in usage can outweigh the lower unit rate.

Inference follows that pattern when teams add AI to more workflows, increase context windows, introduce multi-step agents, or run models against tasks that previously received no automation. Lower rates make those decisions easier to approve, but they do not prove that the new consumption creates value.

That is why falling token rates are not, by themselves, a cost strategy. They are a tailwind that you capture only if you can see where usage is growing and why. Unit cost is the more durable management measure because it connects consumption with the feature, customer, or task receiving the benefit.

Ask a harder question than “What did we spend?” Ask: What did each unit of inference accomplish? Without that connection, aggregate usage can look like product momentum while masking workloads with weak economics.

What did ProjectDiscovery change about its inference economics?

ProjectDiscovery’s documented result shows that prompt packaging can materially affect inference spend without changing models, features, or traffic. Its initial cache hit rate was 7% because dynamic working memory sat inside the system prompt and invalidated the cacheable prefix on nearly every step.

The team moved that dynamic working memory out of the system prompt, raising the cache hit rate to 74% in one deployment. It then added explicit cache breakpoints and deliberate time-to-live settings, reaching 84%.

Before scaling usage, examine the mechanics behind the bill:

• Are repeated instructions and context eligible for caching?
• Does changing working memory invalidate a reusable prefix?
• Are outputs longer than the use case requires?
• Is reasoning enabled where its additional work has not been assessed?
• Can non-urgent workloads use batch processing?
• Does a multi-call workflow produce an outcome that warrants its cumulative consumption?
  This is governance, not merely optimisation. Teams need to know which architectural choices change consumption, which outcomes justify them, and who can stop or redesign a workload when the economics no longer hold.

How should leaders use an outcome ledger before scaling?

Start with the workload, not the provider catalogue. Record the input and output consumption, caching behaviour, reasoning mode, call count, and business purpose for each important AI feature.

The ledger should instead support a decision: scale, redesign, hold, or stop. That decision becomes defensible when finance, product, engineering, and operations can see both sides of the equation – what inference costs and what the workload delivers.

Key takeaways

• Review unit cost by feature, customer, or task, not only total token consumption.
• Treat input and output as separate cost drivers; generated output is typically priced higher.
• Track cached input, reasoning tokens, and batch usage as distinct billing categories.
• Investigate prompt and memory structure before assuming a cheaper model will solve the problem.
• Scale only when the workload’s delivered outcome supports its recurring inference consumption.

Practical tips

• Keep the input and output token counts separate in operational reporting; combining them hides the more expensive generated work.
• Add cache hit rate and cache invalidation behaviour to workload reviews, especially when system prompts include changing working memory.
• Record whether reasoning mode is enabled so an unexplained increase in billed output can be investigated.
• Classify workloads by response urgency; batch processing may be relevant for work that does not require immediate answers.
• When reviewing a cost-reduction result, compare the workload’s models, features, traffic, and architecture so savings are not attributed to the wrong lever.

Make AI consumption reviewable

Use an outcome ledger to give engineering, product, finance, and operations a shared view of what each AI workload costs and what it delivers.

AI MODIFIED