Tokens and context windows

What is a token?

A tokenizer maps text to numbers the model can process. A token might be a whole word, a piece of a word, punctuation, or a space. Each model family can split the same text differently. Exact counts for a particular input come from that model’s tokenizer or the runtime’s reported counts.

The model reads the input token sequence and predicts a next token. It adds that token to the sequence and repeats. That is why a reply can appear a piece at a time.

Budget the context window

The context window is the amount of information a model can use for a request. Instructions, message history, document excerpts, and generated text all draw on that budget. The runtime’s configured limit can be smaller than the maximum listed on the model card.

Suppose your runtime allows 4,096 tokens. If instructions, history, and excerpts use 3,400, roughly 696 remain before accounting for any chat-template overhead. If you need a 1,000-token answer, shorten the material or increase the configured context if memory allows.

Many runtimes keep intermediate attention values in a KV cache. Longer sequences usually grow that cache, so increasing context can increase memory use even with the same model file.

Try it

1. Start a fresh chat and give it a made-up fact, such as “The archive opens at 10.” Ask it a few unrelated questions, then ask for the archive time.
2. Start another fresh chat and ask for the archive time without supplying the fact. Compare the result.
3. In an app with a context setting, try a smaller context with a long piece of sample text. Record whether your runtime truncates input, drops history, or reports an error.

Check your understanding

A model advertises 32,000 tokens, but your runtime is configured for 4,096. Which limit governs this request?

Start with the runtime’s effective 4,096-token limit and leave room for the reply.

Sources

• Hugging Face LLM Course · Tokenizers
• Ollama · Context length
• Hugging Face Transformers · KV cache