Products & Platforms LLM & Generative AI Infra & Hardware

inference cost

Inference cost

Inference cost is the ongoing cost of running data through a trained AI model to produce an output—like a prediction, generated text, or a classification. In large language models (LLMs), both the prompt you send and the tokens the model generates consume compute, so costs scale directly with usage, token volume, and context length.

Difficulty

Plain Explanation

There was a budgeting problem: teams spent a lot to train an AI model once, but then were surprised when monthly bills kept climbing as more people used the model. This concept—Inference cost—solves the confusion by making it clear that you pay every time the model answers a request, not just when you train it.

Think of it like running a printing shop. Buying the printer (training) is a one-time purchase. But ink and paper (inference) cost money every time someone prints. The busier your shop, the higher your ongoing bill.

Why it works this way: each user request gets turned into units called tokens that the model processes. In LLMs, every token in your prompt and every token in the model’s response requires computation. More tokens, longer context windows, or larger models mean more compute per request—so the bill rises with usage and complexity.

Example & Analogy

• Legal document triage: A firm uploads long contracts to summarize risks. Longer prompts (more pages) and longer summaries mean more tokens per request. As more clients start using the service daily, the month-end bill grows linearly with the number of documents processed.

• Customer support deflection: A help center routes user tickets through an LLM that proposes answers. During product launches, ticket volume spikes, doubling API calls. Even if cost-per-call is small, the surge in queries turns into a noticeable jump in monthly spend.

• Real-time meeting notes: A conferencing tool generates live summaries and action items. Because conversations are long, the context window is large, and costs accumulate with every additional minute added to the prompt and every word generated in the output.

• A/B testing prompts: A product team tests multiple prompt templates to improve answer quality. Running the same question across several variants multiplies token usage. If each variant also requests longer, more detailed outputs, inference cost grows faster than expected.

At a Glance

	Large model (70B+ parameters)	Small model (~7B parameters)	Long-context prompts
Per‑query compute	High (more tokens cost more to process)	Lower (cheaper per token)	High (large prompt windows increase tokens)
Typical cost impact	Can be 10x or more per query vs small models	10x cheaper per query vs very large models (directionally)	Costs grow with context length even on the same model
When it matters	Highest accuracy goals, complex reasoning	Cost-sensitive, high-volume apps	Summarizing long docs, chat history retention
Scaling risk	Expensive at high traffic	Easier to scale usage	Invisible creep: longer prompts silently raise spend

Why It Matters

If you ignore token counts, your pilot can look cheap but production will explode in cost as requests and context lengths grow.
Choosing a model that’s larger than needed can multiply per‑query costs by 10x or more, making your unit economics unworkable.
Skipping measurement of prompt and output lengths hides the real driver of spend; you lose the ability to predict monthly budgets as adoption increases.
Underestimating inference’s share of total AI cost leads to bad planning: most organizations end up spending the majority of their AI budget on inference, not training.

Where It's Used

• LLM API pricing and deployment: Providers increasingly compete on lower-cost inference for language models. Some providers (e.g., DeepSeek and low‑price APIs from ByteDance, Alibaba, Baidu, Tencent) emphasize lower per‑request costs while maintaining performance. • Enterprise production rollouts: Companies discover that inference—recurring, per‑request compute—dominates long‑term cost as user traffic scales. • Edge to data center planning: As AI moves beyond centralized servers, feasibility is often defined by whether continuous inference can be delivered efficiently within energy and compute limits.

▶ Curious about more?

Role-Specific Insights
What mistakes do people make?
How do you talk about it?
What should I learn next?
What to Read Next

Role-Specific Insights

Junior Developer: Instrument your app to log prompt and output token counts per request. Set up simple dashboards so you can see how changes in prompts or features affect cost the same day. PM/Planner: Define unit economics early: target cost per successful answer. Limit context length and output size in specs, and plan fallbacks (e.g., shorter summaries) when traffic spikes. Senior Engineer: Choose model sizes by workload. Use smaller models for the majority path and route only hard cases to larger models. Monitor p50/p95 token usage and enforce caps. Finance/Leadership: Treat inference as an ongoing operating expense that scales with adoption. Require cost projections at 1x, 10x, and 100x traffic to avoid surprises.

Precautions

❌ Myth: “Training is the expensive part; inference is cheap.” → ✅ Reality: Training is a one‑time cost, but inference runs on every single request and usually becomes the main cost center in production. ❌ Myth: “Prompt tokens are free; you only pay for generated output.” → ✅ Reality: In LLMs, both prompt and output tokens consume compute and drive cost. ❌ Myth: “If cost per query goes down over time, we’re safe.” → ✅ Reality: Usage often grows faster than per‑query price drops, so total monthly spend can still surge. ❌ Myth: “Only model size matters.” → ✅ Reality: Model size, token volume, context length, and infrastructure/energy overhead all contribute to inference cost.

Communication

• Finance to Platform team: “At $0.01 per query, the pilot looked fine. But at 1M queries/month that’s ~$10k. If marketing hits their goal, 10M queries puts us near $100k monthly—do we have a plan to cap context length?” • ML Ops stand‑up: “Token logs show average prompt length jumped 2x after the new template. Output length also grew. Let’s roll back and A/B test a shorter system prompt to bring inference cost down.” • Product review: “The 70B model performs best, but our budget can’t handle it at peak traffic. Can we ship the 7B model for most queries and escalate only complex cases?” • Architecture discussion: “Inference is our steady-state workload. Let’s design around continuous inference efficiency rather than treating optimization as an afterthought.” • Quarterly planning: “Most of our AI spend is inference, not training. Cost controls will focus on token budgets, context window limits, and model selection.”

Related Terms

Training cost — One-time spend to build the model vs. inference’s recurring per‑request cost; knowing the split helps budget realistically as usage scales.
Tokens — The atomic units LLMs process; both prompt and output tokens drive cost, so trimming tokens often cuts spend quickest.
Model size (parameters) — Larger models can deliver higher quality but may cost 10x+ more per query; right‑sizing saves money without always losing accuracy.
Context length — Longer prompts increase tokens and cost; managing history windows and document chunking is a direct lever on spend.
Inference efficiency — Operational focus on reducing compute per response; small gains matter at scale because they apply to every request.
Compute resources and energy — Under the hood, hardware and power usage shape the real cost floor for sustained inference workloads.

0to1log Weekly

AI Glossary