Cost per Token

Cost per token is the standard pricing metric for LLM inference services, measuring how much it costs to process or generate a single token (roughly ¾ of a word in English). It is the fundamental unit of economics for deploying and using large language models at scale.

Typical Pricing Structure

• Input Tokens: Charged at a lower rate — these are the tokens in your prompt that the model reads and processes.
• Output Tokens: Charged at a higher rate (often 2–4× input cost) — these are the tokens the model generates in its response.
• Example: OpenAI's GPT-4o charges approximately $2.50 per 1M input tokens** and **$10 per 1M output tokens (as of early 2025).

What Drives Cost Per Token

• Model Size: Larger models with more parameters require more GPU compute per token, directly increasing cost.
• Hardware: The type of GPU or accelerator (H100, A100, TPU v5) and its utilization rate heavily influence unit economics.
• Batch Size: Higher batch sizes improve GPU utilization and reduce cost per token through better throughput.
• Quantization: Running models in INT8, INT4, or FP8 precision reduces memory and compute requirements, lowering cost.
• Infrastructure: Self-hosted vs. cloud API pricing reflects different overhead structures.

Why It Matters

At scale, cost per token determines whether an AI application is economically viable. A chatbot handling millions of conversations per day can spend thousands of dollars per hour on inference. Optimizing cost per token through model selection, quantization, caching, and efficient batching is a critical engineering challenge.