Tokens. The Double-Edged Sword.

Every team I have worked with building on top of language models learns about tokens twice. Once when they read the documentation. Once when they see the invoice. The second lesson is usually more memorable, and it is avoidable if you understand what tokens actually are and why they impose constraints on both cost and output quality at the same time.

I have watched this play out with enough clients to know when it is coming. Workflows consuming 40,000 tokens per call when 8,000 would do the same job. System prompts that grew by 50% during development and were never pruned back. Context loading entire documents when only three paragraphs were relevant. The billing surprise is not random. It is the compound interest on a hundred small decisions that were not made with tokens in mind.

What a token actually is

A token is not a word. It is not a character. It is a chunk of text that the model's vocabulary divides language into using a technique called Byte Pair Encoding, or BPE. The algorithm starts with individual characters and iteratively merges the most frequent pairs into single tokens until it reaches a target vocabulary size. The boundary falls in places that feel arbitrary until you understand the underlying encoding. A rough rule of thumb: one token equals approximately 0.75 words in English. One thousand tokens is around 750 words of readable text.

Think of tokens as postage stamps on text. Every word you send to the model and every word it sends back requires stamps. Your system prompt, the user's question, every document you pull into context, the full response, each one is metered and billed. The analogy breaks down in one place: unlike stamps, output tokens cost three to five times more than input tokens. The letters the model writes back cost significantly more than the ones you send, which is why most budget surprises come from output-heavy workflows, not input-heavy ones.

Common words like "the" and "is" are single tokens. Longer or less common words split into multiple tokens. "Hamburger" tokenizes as one unit. "Hamburgers" becomes two. Technical terminology, code, and non-English text often tokenize less efficiently than standard English prose, meaning you consume more tokens per meaningful unit of content. A 10-page proposal document is roughly 3,500 to 4,000 tokens. At current input prices, a single call with that document costs fractions of a cent. Run it through a pipeline 10,000 times per month and the math changes. Count tokens at design time using OpenAI's tiktoken library before you commit to a prompt architecture. As Simon Willison documented in his analysis of GPT tokenizers, the tokenizer exhibits a clear English preference, Spanish text requires roughly 60% more tokens than its English equivalent, and non-Latin scripts like Japanese face even worse efficiency with individual characters splitting into multiple tokens.

What the major providers charge right now

Pricing as of June 2026, sourced directly from provider documentation:

OpenAI's current flagship is GPT-5.4 at $2.50 per million input tokens and $15 per million output tokens, with a 1M-token context window. GPT-5.5 moves to $5 per million input and $30 per million output for their highest-capability tier. These are per-million numbers, a single 10,000-token call costs $0.025 on GPT-5.4.

Anthropic's Claude Opus 4.8 prices at $5 per million input tokens and $25 per million output tokens, with a 1M-token context window. Claude Sonnet 4.6, the model that hits the best cost-to-capability ratio for most production workloads, runs $3 per million input and $15 per million output, also at 1M context. Claude Haiku 4.5 is the budget tier at $1 per million input and $5 per million output. For reference, Claude Opus 4.1 charged $15 per million input and $75 per million output before its deprecation, those numbers show exactly how fast the pricing floor has dropped.

Google's Gemini 2.5 Flash is the cost-efficient choice for high-volume workloads at $0.30 per million input tokens and $2.50 per million output tokens. Gemini 2.5 Pro tiers by prompt length, $1.25 per million input for prompts under 200K tokens, $2.50 per million for prompts above that threshold, and $10 to $15 per million output depending on context size.

The trajectory matters as much as the current number. In early 2023, GPT-4 input tokens cost roughly $30 per million. Today's frontier models deliver substantially higher capability at a fraction of that price, what costs $5 per million today cost 10 to 15 times more two years ago. That curve is not slowing down, which is exactly why instrumenting your token usage from day one is not optional.

The hidden multiplier on output tokens

Output tokens cost more than input tokens on every major provider. The ratio runs from 5x on Claude Sonnet 4.6 ($3 input vs $15 output) to 6x on GPT-5.5 ($5 input vs $30 output). This asymmetry is the most common source of billing surprises, because most teams model their cost estimates on input token volume without accounting for what the model writes back.

A prompt that produces a long structured JSON response can triple the expected cost of a call. If you are generating reports, extraction outputs, or any structured data format, count the output tokens not just the input. A pipeline that sends 8,000 tokens of context and receives 4,000 tokens of JSON output is not an 8,000-token call, it is a 12,000-token call where 33% of the volume costs 5x more per token than the rest.

Structured output and JSON mode do not inherently increase token count, but they constrain the model's output format in ways that often cause it to produce more verbose responses than necessary. Set explicit output length constraints in your prompt and validate that the response structure matches what you actually need. A system that returns a 200-token structured response where 2,000 words of explanation would also satisfy the requirement is saving 90% of output token cost on every call.

Prompt caching changes the math for repeated calls

Both Anthropic and OpenAI now offer prompt caching, and it fundamentally changes the economics of any pipeline that sends the same prefix repeatedly.

Anthropic's implementation uses a cache_control header on content blocks. Mark a prefix as cacheable and Anthropic stores it for a 5-minute TTL by default, or 1 hour at additional cost. The pricing for Claude Opus 4.8 works as follows: cache writes cost $6.25 per million tokens (1.25x the base input price). Cache reads cost $0.50 per million tokens, which is 10% of the base input price of $5. That 10x reduction on reads is what drives the economics for repeated-call pipelines.

OpenAI's caching is automatic on prompts over 1,024 tokens across all recent models. No code changes required. Cached portions see up to 90% cost reduction and up to 80% latency reduction according to OpenAI's documentation. Cached prefixes typically stay active for 5 to 10 minutes of inactivity, with up to 1-hour retention and extended 24-hour retention on newer GPT-5 models. There are no additional fees for the caching mechanism itself.

Here is the math that made this real for me. A RAG pipeline that sends a 50,000-token knowledge base with every query, running 1,000 calls per day on Claude Opus 4.8. Without caching: 50,000 tokens at $5 per million input equals $0.25 per call, or $250 per day. With prompt caching after the first call of each 5-minute window: reads cost $0.50 per million instead of $5, so $0.025 per call, or $25 per day. That is a $225 per day difference, $6,750 per month, on a single pipeline. The break-even point is two calls with the same prefix within the TTL window. Any pipeline running at meaningful volume almost certainly crosses that threshold before lunch.

Caching requires a minimum prefix length to activate. On Claude Opus 4.8 and Sonnet 4.6, that minimum is 1,024 tokens before the cache breakpoint. On OpenAI models, same threshold. For most production system prompts this is easy to reach, but verify using the usage fields returned in the API response, cache_creation_input_tokens and cache_read_input_tokens on Anthropic will tell you exactly whether caching is firing on each call.

Why the same text costs more with some providers than others

Different providers train their tokenizers on different corpora with different vocabulary sizes, which means the same input text produces a different token count on GPT-5.4, Claude Opus 4.8, and Gemini 2.5 Flash. You cannot assume that a prompt costing $X on one provider will cost the same on another.

A practical example from Anthropic's own documentation: the Claude Opus 4.7 generation introduced a new tokenizer that produces roughly 30% more tokens from the same text compared to models before Claude Opus 4.7. That means the same system prompt that cost $0.10 on an older Claude model costs $0.13 on a newer one, not because the price per token changed, but because the token count did. This catches teams off guard when migrating between model versions.

Code, markdown, and technical content tokenize less efficiently than plain prose. A system prompt with heavy markdown formatting, headers, bullet lists, code blocks, can cost 15 to 20% more than the same content written in plain paragraphs. During prototyping I often draft system prompts in markdown for readability, then flatten them to plain text for production once the behavior is dialed in. The formatting that helps a human read the prompt does not help the model process it, and it costs tokens on every single call.

To count tokens accurately before committing: use OpenAI's tiktoken library for GPT models or Anthropic's token counting endpoint before submitting a prompt. The community tool at tiktokenizer.vercel.app provides a browser interface for testing OpenAI tokenization without code. The count is not intuitive and the math matters at scale.

The context window constraint

The context window is the maximum number of tokens a model can process in a single call, covering everything: input, output, and any conversation history. Larger context windows are marketed as a pure capability improvement. They are, but they carry a quality trap that the marketing does not mention.

Research consistently shows that model attention is not uniform across a long context. A landmark study on long-context attention from Stanford documented this directly: content buried in the middle of a long context receives measurably less reliable attention than content at the beginning or the end. Content buried in the middle of a 200,000-token context is processed, but the model is statistically less likely to weight it correctly when forming its output. The practical implication: padding your context with loosely relevant information does not just cost money. It actively degrades output quality by diluting the signal with noise the model partially ignores and partially misweights.

The current generation of 1M-token context windows, available on Claude Opus 4.8, Claude Sonnet 4.6, GPT-5.4, GPT-5.5, Gemini 2.5 Flash, and Gemini 2.5 Pro, makes it technically possible to load entire codebases, full research papers, or extensive document archives into a single call. Whether that is economically rational depends on your use case. Gemini 2.5 Pro tiers its pricing at the 200K-token threshold, charging double the input price for prompts above 200K tokens. That pricing structure is a direct signal from Google that larger context comes at higher compute cost, not just higher billing cost.

The double edge lands here. More tokens in the context window can mean better results, or it can mean worse results at higher cost, depending entirely on whether every token in that window is earning its place. The analogy I use with clients: context is not a filing cabinet where more storage means more capability. It is a whiteboard. A whiteboard with more space is useful, but only if you use that space to write the right things. Fill it with loosely related material and the person reading it has to work harder to find what matters. The model has the same problem, and unlike a person, it cannot ask you to erase the irrelevant parts.

Token efficiency as a design principle

The teams getting the best return from language models treat the context window like expensive real estate. Every token has to earn its place. That discipline produces systems that cost less and perform better simultaneously, because they are eliminating the tokens that add cost without adding signal.

Practical techniques that compound quickly: chunk retrieved content to the sections actually relevant to the current query rather than feeding entire documents. Compress system prompts after they stabilize, removing instructional scaffolding that was useful during development but is now boilerplate. Implement prompt caching for any static prefix over 1,024 tokens that you send on every call. Set explicit output length constraints in the prompt so the model does not produce verbose responses when a concise one serves the use case.

None of these are exotic optimizations. They are basic hygiene, and most production pipelines do not implement any of them in the first version.

What to measure before you scale

Every major AI API returns token counts in the response object. Input tokens used. Output tokens generated. Cached tokens read. Cached tokens written. This data is available on every single call and requires no additional tooling to capture. Log it from the first day you go to production.

With 30 days of token logs you can identify which workflows consume disproportionate tokens and which system prompts have accumulated redundancy. Calculate your cost per query before moving to production. A pipeline that costs $0.08 per query sounds fine until you model it at 50,000 queries per month and discover it is a $4,000 monthly line item. That number belongs in the design stage, not the invoice.

The break-even point for prompt caching is two calls with the same prefix. The payback for counting tokens before you build is the first invoice that does not contain a surprise. Cost surprises at scale are a solved problem. They require 10 minutes of setup on day one and the discipline to look at the data before the billing cycle closes.