How do you reduce LLM cost and latency?

LLM cost is driven mainly by the number of input and output tokens and the price of the model, and latency mainly by model size and how many output tokens it generates. The biggest savings usually come from sending fewer tokens, caching repeated prompt prefixes, routing easy requests to smaller models, batching work that is not urgent, and streaming responses.

3 min read·Checked September 25, 2026·Also called LLM cost optimization, LLM latency, inference cost

Read the free chapter →Connection Layer (MCP) guide

You pay for, and wait for, tokens. Nearly every cost and speed improvement comes down to processing fewer of them, processing them on a cheaper model, or not processing them twice.

What drives LLM cost?

Model APIs charge per million tokens, with separate prices for input and output. Output tokens typically cost several times more than input tokens, and larger models cost many times more per token than smaller ones from the same provider. Hidden reasoning tokens from reasoning models are billed as output too.

A quick estimate helps before building: requests per day, multiplied by average input and output tokens, multiplied by the prices. Agents deserve special attention, because each step re-sends the growing history, so a 20-step task can use many times the tokens of a single chat reply.

What drives LLM latency?

Time to first token (TTFT): How long until the first token of the reply arrives; grows with input length and model size
Output speed: Tokens generated per second; smaller models are faster
Output length: The biggest factor for total time, since tokens are generated one after another
Round trips: Each extra model or tool call in a chain adds its full latency
Queueing: Busy periods and rate limits can add delay beyond the model itself

Users perceive time to first token most, which is why streaming the response as it is generated makes an application feel much faster even when total time is unchanged.

What are the most effective ways to reduce cost and latency?

Technique	Saves	How
Send fewer input tokens	Cost, TTFT	Trim system prompts, retrieve fewer and better chunks, drop old history; see context engineering
Ask for shorter outputs	Cost, total latency	Request concise answers and set a sensible maximum output length
Prompt caching	Cost, TTFT	Put stable content first so repeated prefixes are billed at a steep discount
Use a smaller model	Cost, latency	Test whether a cheaper model passes your evals for each task
Route by difficulty	Cost	Send easy requests to a small model and hard ones to a large model
Batch APIs	Cost	Anthropic and OpenAI offer asynchronous batch processing at about half price for work that can wait
Response caching	Cost, latency	Store answers to repeated identical questions and skip the model entirely
Parallelise	Latency	Run independent calls, such as checks or subagent tasks, at the same time
Stream	Perceived latency	Show tokens as they arrive
Lower reasoning effort	Cost, latency	Use the smallest thinking budget that passes your evals

How do you cut cost without hurting quality?

Measure quality on the same eval set before and after every cost change. Most cost work is a trade-off, and without evals it is easy to save 40% and lose quality you did not notice.

1
Measure
Log input, output and cached tokens, cost and latency for every call, tagged by feature.
2
Find the big spenders
Usually a few prompts or agent flows account for most of the bill.
3
Fix waste first
Remove unused context, oversized outputs and duplicate calls; these save money with no quality risk.
4
Cache
Restructure prompts so stable parts come first and are cached.
5
Downsize carefully
Try a smaller model or lower reasoning effort, and keep the change only if evals hold.
6
Re-check
Prices and models change often; review the numbers every few months.

Do: Track tokens and cost per feature from the first day in production
Do: Set timeouts, retries with backoff, and a fallback model for outages
Do: Put hard limits on agent steps and spending per task
Avoid: Default every request to the largest model
Avoid: Put timestamps or user-specific text at the start of prompts, which breaks caching
Avoid: Optimise latency without looking at time to first token separately from total time

Frequently asked questions

Why are output tokens more expensive than input tokens?

Input tokens are processed together in one parallel pass, while output tokens are generated one at a time, each needing its own pass through the model. That sequential work costs more compute per token.

What is time to first token?

Time to first token (TTFT) is the delay between sending a request and receiving the first token of the reply. It depends mainly on input length, model size and server load, and it shapes how fast an app feels.

Is a smaller model always cheaper overall?

Per token, yes. Overall, not always: if a small model fails and needs retries, longer prompts or human fixes, the total cost can be higher. Compare cost per successful task, not per call.

How much can batch processing save?

Major providers price batch APIs at around half the normal rate, in exchange for results arriving within hours rather than seconds. It suits evaluations, data processing and other work that is not interactive.

Last checked for accuracy on September 25, 2026. Written by the solidcoder team.