A token is a model-readable piece of text
People naturally read sentences as words. A language model reads a sequence of token IDs instead. The tokenizer turns text into those IDs before the model processes a prompt or creates a response.
A token does not have a fixed length. It may represent a frequent word, a word fragment, punctuation, whitespace, a number, or part of a code identifier. That is why a word count can be useful for a rough estimate but cannot replace a real token count.
Why tokens matter in practice
Context capacity
Your instructions, conversation history, source text, and generated answer must fit within a model’s token window.
API usage cost
Many APIs meter input and output separately by the number of tokens, not by the number of words or characters.
Prompt design
Knowing text size helps you trim duplicate context, select a suitable model, and leave room for the response you need.
Tokens vs. words
| Question | Tokens | Words |
|---|---|---|
| What are they? | Tokenizer-created pieces of text. | Human-readable units separated by language conventions. |
| What changes them? | Model encoding, whitespace, punctuation, and text pattern. | Language and how people define a word boundary. |
| What are they used for? | Model limits and API metering. | Reading length and rough planning. |
Ready to count a real prompt?
Use the local calculator to see the token, character, word, and configured API cost reference for your own text.
Open Token Calculator