Tokenization Guide

Tokenization turns a string of text into the numerical pieces an AI model can process. This guide shows why that process affects token counts, limits, and costs.

How tokenization works

Before a language model can read a sentence, a tokenizer matches pieces of the string to entries in an encoding vocabulary. It replaces those pieces with token IDs. The model then predicts the next token ID from the sequence it has received.

Modern text tokenizers often use byte-pair encoding–style vocabularies. They learn common chunks so familiar sequences can be represented efficiently, while uncommon text can still be broken into smaller pieces. Whitespace and punctuation are part of the input, so they can change the result too.

Text is not split by words alone

A leading space, punctuation mark, emoji, URL, or number can become its own token or change the token boundary around it.

Encodings make counts differ

The same text can have a different token total in o200k_base and cl100k_base because their vocabularies are not identical.

Code deserves a real count

Identifiers, indentation, punctuation, and repeated syntax make code especially unreliable to estimate from plain word counts.

A local encoding example

Tokens Calculator ships two browser-safe encodings. They are a practical way to compare text size without a server request.

Tokenization helps AI understand text.

o200k_base

Used for the GPT-4o reference. Its vocabulary can group text differently from older encodings.

cl100k_base

Used for the GPT-4 reference and clearly labeled Claude/Gemini-compatible planning estimates.

Compare an actual text sample

Use the multi-model page to see o200k_base and cl100k_base results side by side, with clear labels for approximation-only rows.

Compare token counts

Use this knowledge in a calculator

01
Choose the closest available encoding
Use GPT-4o for o200k_base or GPT-4 for cl100k_base when measuring text locally.
02
Inspect the actual text count
Paste the original text instead of inferring tokens from characters or words alone.
03
Treat nonmatching models as estimates
For provider-specific model families, use the local result for planning and confirm a production request with the provider’s own tools.

Tokenization Guide FAQ

What is tokenization?

Tokenization is the process that converts text into the smaller numerical units an AI model can process. The model receives token IDs, not a raw sentence of characters.

Why do different models tokenize the same text differently?

Each model family can be trained with a different encoding and vocabulary. Those choices change how frequently occurring words, code, whitespace, and multilingual text are split.

What are cl100k_base and o200k_base?

They are named OpenAI-compatible BPE encodings available in gpt-tokenizer. This site uses cl100k_base for GPT-4 and selected approximation pages, and o200k_base for GPT-4o.

Can a local tokenizer calculate every provider’s exact bill?

No. A local encoding can be a useful planning reference, but exact API billing can also include provider-specific tokenizers, request wrappers, tools, media inputs, and cached content.

Put the guide into practice

Every tool runs token counting in your browser. Keep your text local while you move between calculators and guides.