Cost

LLM pricing and costs are the economic factors that determine the total expense of running AI applications — including API costs per token, self-hosting infrastructure expenses, and optimization strategies, critical for building sustainable AI products and making build-vs-buy decisions.

What Are LLM Costs?

  • Definition: Total expense of using LLMs in production.
  • Components: API fees, infrastructure, optimization, engineering.
  • Unit: Typically cost per million tokens (input and output separately).
  • Variation: 100× difference between cheapest and most expensive options.

Why Pricing Matters

  • Product Economics: AI features must be profitable.
  • Build vs. Buy: Self-hosting vs. API decision.
  • Architecture Choices: Model routing, caching, batching decisions.
  • Scale Planning: Costs compound at scale.
  • Competitive Position: Lower costs enable lower prices or higher margins.

API Pricing Comparison (2024)

Provider/Model          | Input/1M tk | Output/1M tk | Notes
------------------------|-------------|--------------|---------------
GPT-4o                  | $2.50       | $10.00       | Most capable
GPT-4o-mini             | $0.15       | $0.60        | Cost-optimized
GPT-3.5-turbo           | $0.50       | $1.50        | Legacy
Claude 3.5 Sonnet       | $3.00       | $15.00       | Strong reasoning
Claude 3 Haiku          | $0.25       | $1.25        | Fast, cheap
Gemini 1.5 Pro          | $1.25       | $5.00        | Long context
Gemini 1.5 Flash        | $0.075      | $0.30        | Fastest
Llama 3.1 70B (hosted)  | $0.20-0.80  | $0.20-0.80   | Varies by host
Mistral Large           | $2.00       | $6.00        | European option

Self-Hosting Economics

Infrastructure Costs:

Hardware Option    | Monthly Cost | Models Served
-------------------|--------------|--------------------
RTX 4090 (24GB)    | ~$500 amort. | 7-13B models
A100 40GB          | $2-3K cloud  | Up to 30B
A100 80GB          | $3-4K cloud  | Up to 70B
H100 80GB          | $4-6K cloud  | 70B+ fast inference
8× H100 cluster    | $30-40K      | Any model, high throughput

Break-Even Analysis:

API cost example: $5/M tokens × 10M tokens/day = $50/day = $1,500/month
H100 cost: ~$5,000/month
Break-even: ~100M tokens/day for H100
Below this: API often cheaper
Above this: Self-host saves money

Cost Optimization Strategies

Caching:

Common queries → Cache responses
Hit rate of 20% → 20% cost reduction
Semantic caching: Similar queries hit cache
Implement: Redis, custom cache layer

Model Routing:

Simple queries → Cheap/small model (90% of traffic)
Complex queries → Expensive/large model (10% of traffic)
Potential savings: 60-80%

Prompt Optimization:

Before: 2,000 token system prompt
After: 500 token optimized prompt
Savings: 75% on input tokens

Techniques:
- Compression
- Remove redundancy
- Batch instructions

Output Control:

max_tokens: Set appropriate limits
Stop sequences: End early when possible
JSON mode: Structured output (often shorter)

Batching:

Real-time: Process individually (higher per-request cost)
Batch: Accumulate, process together (lower per-request cost)
When acceptable latency allows, batch for savings

Cost Tracking

What to Measure:
- Tokens per request (input + output).
- Requests per user/feature.
- Cost per user action.
- Cost per successful outcome.

Implementation:

class CostTracker:
    def __init__(self):
        self.costs = defaultdict(float)
    
    def record(self, user_id, feature, 
               input_tokens, output_tokens, model):
        cost = calculate_cost(
            input_tokens, output_tokens, model
        )
        self.costs[user_id] += cost
        self.costs[feature] += cost
        self.log(user_id, feature, cost)

Cost by Use Case

Use Case              | Typical Cost      | Optimization
----------------------|-------------------|-------------------
Chat (1 turn)         | $0.001-0.01       | Cache, small model
Code completion       | $0.0001-0.001     | Small model, prefix caching
Document summary      | $0.01-0.10        | Batch, smaller model
RAG (search + answer) | $0.005-0.05       | Cache embeddings
Agent (multi-step)    | $0.10-1.00        | Limit retries, cheaper tools

Cost Control Architecture

LLM Cost Optimization — Dollars per Million Tokens model choice × quantization × caching × routing × batching = 10-100× cost reduction at equal quality API Pricing ($/1M output tokens, July 2026) Haiku $1.25 GPT-4o-mini $0.60 Sonnet 3.5 $15 GPT-4o $10 Opus $75 o1-pro $150 Self-hosted (vLLM on H100) Llama-3 70B: ~$0.40/MTok at full utilization Llama-3 8B quantized: ~$0.05/MTok Output tokens cost 3-5× more than input tokens (decode is memory-bandwidth bound) Cost Reduction Levers 1. Model routing (5-10×) cheap model for easy queries 2. Prompt caching (2-5×) reuse system prompt KV cache 3. Batch API (2×) async processing at 50% discount 4. Prompt compression (1.5-3×) fewer input tokens (LLMLingua) 5. Self-host + quantize (10-50×) open model, INT4, full GPU util 6. Distillation train small model on large's outputs Cost Math Example Scenario: 1M requests/day, 500 in + 200 out tokens avg GPT-4o: $2.5K input + $2K output = $4.5K/day GPT-4o-mini: $75 input + $120 output = $195/day Self-hosted 8B: ~$35/day (1× H100 at $1.50/hr) Saving: $4.5K → $195 (route) → $35 (self-host) = 130× cost reduction, quality depends on task Optimization Strategy by Stage Day 1: Prototype use best model, ignore cost establish quality baseline Week 2: Evaluate test smaller models on eval set find cheapest at same quality Month 1: Route classify queries, route to tiers 80% queries → mini model Month 3: Self-host fine-tune open model, deploy maximize tokens/$/quality The frontier model is the eval baseline, not the production model — always ask: can a cheaper model do this task? LLM cost optimization is the new cloud cost optimization — the companies that solve it first win on unit economics.

LLM pricing and costs are the foundation of AI product economics — understanding and optimizing costs determines whether AI features are sustainable at scale, making cost engineering as important as prompt engineering for production AI systems.

Take cost further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app