Your language may cost more tokens than mine

Tokenisation is where the English-first assumption stops being philosophical. A tokenizer learns to split text into pieces based on what it saw most. Trained on a corpus that skews Latin-script, it learns long, efficient pieces for English words and short, inefficient ones for everything else. Devanagari, Thai, Korean and many others end up fragmented into far more tokens for the same sentence.

Three consequences, none of them abstract

• Cost: you are billed per token, so the same request costs more in some languages than others.
• Context: a fixed context window holds less of your actual content, so long documents get truncated sooner.
• Latency: more tokens to generate means a slower response, on exactly the connections that are already slower.

The uncomfortable part is that this is invisible to the person paying it. Nobody sees a line item that says their script is more expensive to think in. They just experience a tool that feels slightly worse and cannot say why.

What can actually be done

Some of this is ours to fix and some of it is not. We do not train the frontier models, so we cannot rewrite their tokenizers. What we can do is stop compounding the problem: keep prompts short in the user's language rather than padding them with English scaffolding, cache aggressively at the structure level so the same layout is not re-derived every time, and do work in code that does not need a model at all.

We also measure it. For every language we support we track tokens-per-build and time-to-first-paint, and we look at the gap between the best and worst languages rather than the average. An average is how a long tail gets hidden.

If the gap for your language looks wrong, tell us what you typed. That report is worth more than a benchmark, because it comes with the sentence a real person actually wanted to write.