Sam Altman’s point is real. AI token costs have suddenly become a serious enterprise problem. At OpenAI’s Intelligence at Work event, he said AI budgeting went from barely coming up earlier this year to becoming “a huge issue,” and said OpenAI’s top token user burns around 100 billion tokens per month.
That number sounds absurd until the mechanics are understood. Tokens are not just what appears in the user’s prompt. Tokens are consumed when the model reads, reasons, responds, checks, retries, and processes context. The visible chat box is only the surface area. Behind it sits system instructions, memory, retrieved documents, tool outputs, chat history, safety instructions, OCR text, transcripts, and generated responses.
The token crisis is usually framed as a usage problem. More people are using AI, therefore more tokens are being consumed. That is true, but incomplete. A large part of the token crisis is a quality problem. The worse the model performs, the more users have to steer, correct, clarify, and retry. Every failure becomes another billable interaction.
What exactly is a token?
A token is a unit of text that a language model can process. It is not always a full word. It can be a whole word, part of a word, punctuation, whitespace, or a common sequence of characters. In English, a rough mental model is that one token is about three to four characters, or around three-quarters of a word, but the exact number depends on the tokenizer.
For example, using OpenAI’s common cl100k_base tokenizer, “please” is one token, “sorry” is one token, “thank you” is two tokens, “Thanks!” is two tokens, “Please and thank you.” is five tokens, and “Sorry, can you try again?” is seven tokens.
This is why politeness became a funny but revealing example. One “thank you” costs almost nothing. Two tokens are meaningless at the individual level. But at global scale, tiny phrases become real infrastructure load. Sam Altman has said that people saying please and thank you to ChatGPT costs OpenAI tens of millions of dollars. The point is not that manners are expensive in one conversation. The point is that billions of tiny, unnecessary exchanges compound across hundreds of millions of users.
Tokens are not exactly compute, but they are closely tied to compute. A model does not “think” in words the way humans do. It receives a sequence of token IDs, processes those through the network, and predicts the next token. More input tokens mean more context to process. More output tokens mean more generation steps. Longer conversations mean the model may need to carry more history forward. In transformer models, context length can become especially expensive because the model has to manage relationships between tokens across the sequence.
So the token became the commercial unit because it is measurable, model-native, and reasonably correlated with cost. It is not a perfect compute meter. Different models have different architectures, different inference stacks, different caching, different hardware efficiency, and different media-processing costs. But tokens are the closest simple billing unit for text-based AI because they measure the actual material the model consumes and produces.
Who decided on the token economy?
There was no central authority that decided the AI economy would run on tokens. The measurement came from decades of natural language processing, where text had to be broken into machine-readable units before models could learn from it. Early systems used characters, words, subwords, or handcrafted vocabularies. Modern LLMs mostly use subword tokenization because it balances efficiency with flexibility.
A major step was byte pair encoding, or BPE, adapted for neural machine translation by researchers such as Rico Sennrich, Barry Haddow, and Alexandra Birch in their 2016 work on rare words and subword units. The idea was simple but powerful: instead of forcing the model to choose between characters and whole words, text could be split into frequent subword chunks. Common words could stay compact. Rare words, names, slang, code, and new terms could still be represented by smaller pieces.
GPT-style models adopted this family of tokenization approaches. OpenAI then made token-based pricing mainstream through its API model. Other providers followed because tokens gave developers a predictable way to estimate cost, compare models, set rate limits, and bill usage.
So the answer is: researchers created tokenization because models needed a practical way to represent language, and AI providers turned tokens into an economic unit because they needed a scalable way to meter usage.
Why bad model quality makes token consumption worse
The missing piece in most token-cost conversations is model quality. If a model gives a useful, grounded answer in one exchange, token usage stays contained. If the model gives a vague, generic, wrong, or hallucinated answer, the user has to spend more tokens fixing it.
That is why bad data matters. If training data is weak, polluted, synthetic, repetitive, or recursively generated by other AI systems, model quality deteriorates. The model may still sound fluent, but it becomes less grounded. It gives answers that feel plausible and require human correction. Research on model collapse has shown that recursively training generative models on model-generated data can degrade the tails of the original data distribution and damage reliability.
This has a direct token cost. The worse the answer, the longer the prompt becomes. Users start writing defensive instructions. They paste more examples. They explain what not to do. They ask for citations. They correct hallucinations. They ask for rewrites. They say “no, that’s not what I meant.” They turn a simple task into a negotiation with the machine.
That negotiation is paid for in tokens.
This is why token consumption is not just an infrastructure problem. It is a product-quality problem. A bad AI system does not only frustrate the user. It makes the user pay for the frustration through repeated prompts, longer prompts, larger context, and more correction loops.
The real cost of hallucination
Hallucination has a direct economic footprint. Every fake citation, invented feature, bad assumption, wrong summary, or broken piece of code creates follow-up work. In a consumer chat, that means another message. In an enterprise workflow, it can mean another retrieval pass, another agent loop, another human review, another compliance check, another escalation, and another generated report.
The bill is not just inference. The bill is failure recovery.
This is why the industry’s obsession with bigger context windows and more autonomous agents is dangerous if the underlying reliability problem is not solved. Bigger windows let users shovel in more context. Agents let systems loop longer before humans notice. Multimodal inputs let more of the world become machine-readable. All of that can be useful, but all of it can also multiply waste.
The token crisis is not only caused by people using AI more. It is caused by AI still needing too much supervision to produce reliable outcomes.
The practical takeaway
Tokens represent the material an AI system reads and writes. They are not a perfect measure of compute, but they are the dominant billing and capacity unit because they map closely enough to model workload. Long prompts, long answers, repeated corrections, agents, retrieval, OCR, voice, video, hidden instructions, and bad-quality outputs all increase token consumption.
The smallest waste is saying “thank you.” The largest waste is needing to say “try again” ten times.
The future of AI cost reduction is not just cheaper inference. It is fewer retries, cleaner data, better retrieval, stronger grounding, and models that do not require massive prompting rituals to produce one usable result.
Until those fundamentals improve, token consumption will keep rising because users are not only paying for intelligence. They are paying for the model’s mistakes.
Research notes
OpenAI CEO Sam Altman said AI budgeting became a “huge issue” for customers and referenced an OpenAI top token user consuming around 100 billion tokens per month during the Intelligence at Work event, reported by Tom’s Hardware and Axios.
Goldman Sachs Research has projected that agentic AI could drive a 24-fold increase in token consumption by 2030.
OpenAI’s tokenizer documentation explains that models process text using tokens, which are common sequences of characters found in text.
Sennrich, Haddow, and Birch’s 2016 paper, “Neural Machine Translation of Rare Words with Subword Units,” helped popularize BPE-style subword tokenization in neural language systems.
Nature published “AI models collapse when trained on recursively generated data” in 2024, showing that indiscriminate use of model-generated content in training can cause model collapse and damage reliability.