Individual model calls look almost free — fractions of a cent per thousand tokens. Then the invoice arrives. The reason is structural: an AI agent executes dozens of steps per task, and most agent harnesses resend the accumulated transcript on every turn, so token consumption grows quadratically with session length, not linearly with task complexity. Every tool result, every log line, every retry gets stuffed back into the prompt.
This is a solved engineering problem in 2026, and the fixes are well documented. What follows is the rollout order that delivers the most saving for the least risk.
First, Know Which Costs You Have
Three categories behave differently and need separate tracking.
- Input tokens — the context you feed the model. Grows every turn. Usually the largest and most fixable line.
- Output tokens — typically 3x to 10x more expensive per token than input. Controlled by capping length and structuring responses, not by caching.
- Infrastructure — vector databases, embedding generation, file storage. Scales with your data, not your traffic, and is often the cost nobody attributed to the agent.
If you cannot break your bill into those three, do that before optimising anything.
Step 1: Prompt Caching — the Highest-Leverage Lever
Prompt caching stores the computed key-value tensors behind a repeated prompt prefix, so the static portion of each request — system prompt, tool definitions, reference documents — bills at up to 90% off while the model produces identical output. No quality trade-off. This is why it goes first.
The requirement is prefix stability, which dictates prompt layout:
- Static content first: system instructions, persona definitions, few-shot examples
- Heavy stable context second: the large document or policy set that holds for the session
- Dynamic content last: the user's query and the live conversation tail
Anything that changes per request must sit after everything that does not. A single volatile token — a timestamp, a random session ID — placed near the top invalidates the entire prefix and silently costs you the whole discount.
The workloads where caching pays best are multi-turn conversations (history grows but earlier turns never change, so each request appends a small dynamic tail to a stable prefix), retrieval pipelines that repeatedly pull the same documents or policies, and coding agents.
Step 2: Control the Cache Boundary — Don't Cache Blindly
This is the step most teams skip, and it matters. Research published as Don't Break the Cache (arXiv 2601.06007) tested caching across more than 500 agent sessions with 10,000-token system prompts. Caching reduced API costs by 41–80% and improved time-to-first-token by 13–31% — but the paper's central finding is that naive full-context caching can paradoxically increase latency. Strategic control of where the cache boundary falls outperformed caching everything by default.
The practical rule: a cacheable block only pays if it is genuinely stable. Mark the boundary deliberately at the last point in your prompt that is guaranteed not to change, and measure. Do not set it hopefully at the end of a block that mutates every third turn.
Step 3: Context Hygiene and Compression
Passing the entire conversation history into every call is the default in most frameworks and is almost never correct. Replace it with a rolling window plus periodic summarisation — keep the last N turns verbatim, commonly around five exchanges, and summarise everything older.
Layered with semantic compression, retrieval-based context management and capped reasoning budgets, this reduces cost substantially with minimal measured quality impact. Reported ranges run very wide, from 50% to over 90%, which is itself the lesson: the saving depends entirely on how bloated your baseline was, so measure your own before believing any headline number.
Agent memory deserves specific attention. Traditional memory pipelines run three sequential model calls per new memory — extract, check conflicts, update or merge — which is expensive and adds write latency to every turn. Switching to single-pass, add-only extraction with deferred or asynchronous conflict resolution cuts write-time model calls by roughly 60–70% without meaningfully degrading memory quality.
Step 4: Route by Task Difficulty
Premium reasoning models carry per-token prices 5–10x higher than capable mid-tier alternatives. Sending every sub-task to the best model is the most common and most expensive architectural mistake in production agents.
Two related disciplines:
- Route by difficulty. Classification, extraction, routing decisions and formatting go to a cheap model. Planning and hard reasoning go to the expensive one.
- Be sparing with multi-agent fan-out. Reserve genuine multi-agent architectures for truly independent parallel work. Parallel primary-plus-secondary review patterns can consume 4–15x more tokens than a single well-prompted call, and frequently for no measurable quality gain.
Also worth layering in: batch APIs for anything not user-facing, structured outputs with explicit length caps, and grounding agents with retrieval so you stop pasting whole documents into prompts.
Step 5: Add Semantic Caching on Top
Exact-match caching only helps on identical requests. Adding semantic caching — serving a stored response when a new query is close enough in embedding space — pushes reduction on repeated calls into the 50–90% range. One 2026 paper reported a 41.6% reduction in model API calls (376 cache hits from 903 requests), translating to roughly 40–45% lower operating cost at scale.
Semantic caching is the one technique here that can change answers, so it needs a similarity threshold you have tested and a clear exclusion list for anything time-sensitive or user-specific.
Why It Matters
Agent cost is the main reason promising pilots do not reach production. A demo that costs three dollars per task is thrilling; the same task at production volume is a budget line that finance will kill. Realistic savings with a competent implementation of the above land around 70–80% — enough to move many workloads from unaffordable to routine.
The documented case studies support the ordering. ProjectDiscovery's Neo agent reported a 59% cumulative drop from prompt caching alone, climbing past 90% on fully optimised paths. Start with the provider feature that requires no architectural change, then earn the harder savings.
One standing caveat: cache discounts and per-token rates shift with every model release — DeepSeek's September release alone cut agent memory costs several-fold, and Anthropic cut cache-read pricing by 75% at the start of the month. Confirm current figures against each provider's live pricing and caching documentation before you commit a production budget to them.
