Tokens are the basic units of text that LLM works with. A model does not read text as words in the human sense. Before processing, the text goes through tokenization, the process of splitting a string into smaller elements: words, subwords, whitespace, punctuation, and sometimes even individual characters. These units are the tokens.
It is important to understand that tokenization can be either a whole word or just a fragment of one. That is why the same phrase in different languages, or even in different contexts, can have a different number of tokens.
At the system design level, tokens affect at least four key aspects.
Cost control
In most LLM, the price depends directly on the number of input and output tokens. The longer the prompt, history, retrieved context, or model response, the more expensive each request becomes. One of an AI Engineer's routine tasks is prompt optimization, so it is important to understand how to work with this.
Context window management
Every model has a limit on the maximum number of tokens it can process in a single request. Both the input text and the model's upcoming response count toward this limit. That is why a good AI Engineer thinks about which context provides the most information value per token, in order to make the prompt as short as possible while keeping it as effective as possible.
Latency
The number of tokens directly affects the speed of the system. More tokens mean longer input processing and longer output generation. This is especially critical for real-time chats, copilot products, and similar products where latency directly affects UX.
Quality and reliability
Tokens affect not only price and limits but also the quality of the response. If a prompt is overloaded with irrelevant context, the model focuses worse on the important information. In RAG and agent pipelines scenarios, the engineer's job is to pack the relevant information optimally within the allocated token budget.