What is a token in an AI language model?

A token is a small chunk of text that a language model reads as one unit. It might be a whole word, part of a word, a single character or a bit of punctuation.¹

That's the short answer. The longer one explains why a chatbot's idea of "length" doesn't match yours.

Why models use tokens at all

A computer can't do sums on letters. Before a language model can handle your sentence, the text has to be turned into numbers. One source puts it plainly: machine learning programs process numbers rather than text, so the text must be converted.²

The first step is to agree on a vocabulary, which is a fixed list of chunks. Each chunk gets a number. Your sentence is then cut into chunks from that list, and the model works with the numbers. Those chunks are the tokens.²

Different makers cut text up in different ways. Wikipedia names two common methods, byte-pair encoding and WordPiece. You don't need to know how either works to use a chatbot, but it explains why two products can count the same sentence differently.²

A token isn't a word

OpenAI's help page says a token can stand for a character, part of a word, a whole word or punctuation. It also says a token count is not the same as a word count.¹

So a short, common word is often one token. A long or unusual word, such as a product code or a surname, may be split into several pieces. A comma can be a token by itself.

Spaces matter too. The same page notes that spaces affect how text is divided, and that "red", "Red" and a red with a space in front are not identical text. An encoding can represent each of them differently.¹

That's why you can't convert tokens to words with perfect accuracy. You can only estimate. The next page on word counts gives the makers' rule of thumb for English.

The same text can cost different amounts

The same text can produce different token counts depending on the model, its encoding and the language.¹

Here's what that means for a business user:

• A figure quoted for one assistant may not carry over to another.
• A document in another language may use a different number of tokens than its English twin.
• Two models from the same maker can split the same text differently.

If you need a real number, count the text with the tool for the model you plan to use. How to count tokens shows where those counters are.

Where tokens show up in daily use

You'll meet tokens in three places.

1. Size limits. A model can only take in so many tokens at once. That limit is its context window.
2. Reply length. The text a model writes is made of tokens too, and replies have their own limits. See input and output limits.
3. Price. Many makers charge by the token when you use their tools through a programming interface. See how token-based pricing works.

Most chat apps hide all this. You type words and get words back. But the meter underneath is running in tokens, which is why an assistant can cut off or forget even when your message looked short to you.

A practical way to think about it

Picture a model reading through a slot that's only wide enough for chunks, not whole sentences. Your text goes in as chunks, the model's reply comes out as chunks, and both are counted.

Keep one habit. When a limit or a price is given in tokens, don't treat it as a word count. Treat it as a rough size, and check the maker's own documentation for the model you use.

One more point. Whatever you type gets turned into tokens, so the usual care applies. Secrets such as passwords and keys, and anything about your customers, stay out of a prompt unless a policy at your workplace says that's fine.