LLM Tokens and Context Windows Explained: How AI Models Read, Forget, and Bill Text

November 6, 2026 · 13 min read

Every conversation with a large language model happens inside a limited space. The model cannot see your whole company, your whole website, or even your whole chat history unless that material fits inside a fixed budget called the context window. That budget is counted in tokens, not words or pages. Understanding those two ideas explains a surprising amount of everyday AI behavior: why a long chat seems to "forget" early instructions, why a 200-page contract gets summarized unevenly, why API bills vary so much between similar requests, and why an AI search engine quotes one paragraph of your site and ignores the rest.

This guide explains tokens and context windows in plain English for U.S. B2B SaaS teams. It covers how text becomes tokens, what actually lives inside the window, why bigger windows are useful but not magic, how costs are calculated, and what all of this means for the way AI engines read and describe your brand. If you are new to the topic, start with our plain-English guide to large language models, then come back here.

What a Token Actually Is

A token is the unit of text a model reads and writes. It is usually a piece of a word rather than a whole word. Common short words are often a single token. Longer or rarer words are split into several pieces. Numbers, punctuation, spaces, and code symbols are tokens too.

A rough rule of thumb for ordinary English is that one token is about three quarters of a word, so 1,000 tokens is roughly 750 words. Treat that as an estimate, not a law. The real count depends on the model's tokenizer, the component that splits text into pieces, and different model families use different tokenizers.

Several practical patterns follow from how tokenizers work:

• Plain English is cheap. Everyday vocabulary tokenizes efficiently.
• Jargon, product names, and invented brand words cost more. A distinctive SaaS name may be split into three or four fragments.
• Other languages can cost more. Many tokenizers are trained mostly on English-heavy data, so the same sentence in another language can take more tokens.
• Numbers and code are uneven. Long numbers, IDs, URLs, and dense code often break into many small tokens.
• Formatting is not free. Tables, repeated whitespace, and markup all consume budget.

The model never sees letters the way you do. It sees a sequence of token IDs. That is one reason models sometimes struggle with tasks like counting the letters in a word or reversing a string: the letters are hidden inside tokens.

What the Context Window Is

The context window is the maximum number of tokens a model can consider at once in a single request. Think of it as the model's working desk. Everything relevant to the current answer must be on the desk at the same time. Anything not on the desk does not exist for that answer.

A crucial detail many people miss: the window usually includes both input and output. Your instructions, the documents you paste, the conversation so far, retrieved search results, tool outputs, and the answer being written all draw from the same budget, though many providers also set a separate, smaller cap on output length.

In a typical chat product, the desk contains several layers:

1. System instructions written by the product builder, which you usually cannot see.
2. Conversation history, your earlier messages and the model's earlier replies.
3. Attached or retrieved material, such as uploaded files, web pages, or passages from a company knowledge base.
4. Tool results, if the system can call functions or browse. Our guide to LLM agents and tool use explains how these outputs pile up.
5. The answer in progress, which grows token by token.

When the desk fills up, something has to give.

Why Long Chats "Forget"

Models do not remember a conversation the way a person does. Each time you send a message, the application resends the relevant history as part of the next request. The model reads it fresh every time. Nothing about the conversation is stored inside the model's parameters, a point covered in our article on how LLMs are trained.

When a conversation grows beyond what fits, applications use strategies such as:

• Truncation: dropping the oldest messages.
• Summarization: replacing early turns with a shorter summary, which loses detail.
• Selective retrieval: storing past messages elsewhere and pulling back only the parts that seem relevant.

Each strategy can lose information. If a key instruction appeared forty messages ago, it may have been cut or compressed. The model is not being stubborn; it may simply no longer see it.

Practical fixes are simple: restate critical constraints near the end of long sessions, start a new chat for a new task, and keep a short brief you can paste at the top of a fresh conversation.

Bigger Windows Help, but They Are Not Perfect Memory

Context windows have grown dramatically, from a few thousand tokens in early chat models to hundreds of thousands or more in current models. That lets you hand a model an entire contract, a long research report, or a large code file in one request. It is genuinely useful.

But a large window is not the same as reliable attention to every line. Research on long-context behavior, including the widely cited "Lost in the Middle" study from Stanford and collaborators, found that models often use information at the beginning and end of a long input more reliably than information buried in the middle. Newer models have improved, and results vary by model and task, but the lesson stands: fitting text into the window does not guarantee the model will use it well.

Other limits apply too:

• Cost scales with tokens. Sending 200,000 tokens on every request is expensive.
• Latency grows. Longer inputs take longer to process before the first word of an answer appears.
• Noise dilutes signal. Irrelevant material competes with the passage that actually answers the question.
• Advertised maximums are not always available. A product may expose a smaller window than the underlying model supports, depending on plan or feature.

This is why well-built systems rarely just stuff everything into the window. They select. Our explanation of retrieval-augmented generation covers how systems choose a small set of relevant passages instead of feeding in whole libraries.

How Tokens Drive Cost

Most commercial LLM APIs bill per token, usually quoted per million tokens, with separate rates for input and output. Output tokens are typically priced higher than input tokens because generating text takes more computation per token than reading it.

A simple cost estimate looks like this:

• Input tokens multiplied by the input rate.
• Plus output tokens multiplied by the output rate.
• Plus any extras the provider charges, such as web search calls, tool use, or stored context.

Some details make real bills harder to predict than the simple formula:

• Reasoning tokens. Many current models "think" before answering. Those internal reasoning tokens are often billed as output even when you never see them. A short visible answer can carry a large hidden cost.
• Conversation resending. Because history is resent each turn, a long chat costs more per message as it grows.
• Retrieved content. Web search and document retrieval add the fetched text to the input, which you pay for.
• Caching. Some providers discount repeated identical prefixes, such as a long system prompt reused across requests.

If you are budgeting an AI feature, measure actual token usage on realistic requests rather than estimating from word counts. Set explicit output limits, and treat reasoning and search costs as separate line items to verify, not assumptions.

Output Limits Are Separate From the Window

Models and APIs commonly enforce a maximum output length that is much smaller than the full context window. A model might accept a very long input but cap a single response at a few thousand tokens. When an answer stops abruptly mid-sentence, that cap is a likely cause.

For long deliverables, it is usually better to request work in sections, such as an outline first and then each section, than to ask for one enormous response. Sectioned work is also easier to review for errors, which matters given the risks described in our guide to LLM hallucinations.

Tokens, Windows, and Security

Everything in the window is something the model may act on. That includes text you did not write: a web page the system fetched, an email it summarized, a document a colleague uploaded. If that material contains instructions, the model may treat them as instructions. This is the root of indirect prompt injection, explained in depth in our article on LLM security and privacy.

Two practical rules follow. First, keep sensitive data out of the window unless the task requires it, because anything in the window can appear in the output. Second, do not assume a long system prompt protects you; the model weighs all text on the desk, and attacker-controlled text sits on the same desk.

What This Means for How AI Engines Read Your Brand

This is where tokens and context windows become a marketing issue, not just an engineering one.

When an AI search experience such as Google AI Overviews, ChatGPT search, or Perplexity answers a buyer's question, it typically retrieves a handful of sources and places selected passages from them into a model's context window. The model then writes an answer from that limited material. It does not read your entire website. It often does not even read your entire page. It works from the fragments that were selected and fit.

That has concrete consequences for B2B SaaS companies:

• Your key facts need to be extractable in short passages. If your pricing model, ideal customer, and core capability are only understandable after reading five paragraphs of brand storytelling, the selected fragment may not contain them.
• Put the answer near the question. A page section titled "Who uses [Product]?" should answer in its first two sentences, then elaborate.
• Reduce boilerplate. Navigation text, cookie banners, repeated calls to action, and heavy scripts can crowd out useful content when pages are processed. Server-rendered, readable HTML helps; our guide on structuring a website for LLM crawlers covers this.
• Be consistent across pages. If two passages from your site disagree, for example a pricing page and an old blog post, the model may combine them into a confused answer.
• Date your facts. A passage that says "as of November 2026" carries its context with it, even when it is pulled out alone.

In other words, write as if every paragraph might be read on its own, because inside a context window it often is. This is the practical heart of the difference described in LLM SEO vs. traditional SEO: ranking a page is not the same as having the right paragraph selected and accurately summarized.

Practical Habits for Working Within Token Limits

For everyday users and teams building with LLMs, a few habits go a long way:

1. Lead with the task. State what you want in the first lines, then provide material, then restate the key constraint at the end.
2. Send only what is relevant. Excerpt the three sections of a contract you need reviewed instead of the whole document.
3. Label your inputs. Clear headings like "Source document" and "Instructions" help the model keep roles separate.
4. Ask for structure. Request bullet points, tables, or fixed sections so output stays inside limits and is easy to check.
5. Start fresh for new tasks. A clean conversation avoids stale or conflicting history.
6. Measure, then budget. For API products, log real input, output, and reasoning token counts before committing to a price or volume.
7. Verify claims from long inputs. Ask the model to quote the exact passage supporting each important claim, then check the quote exists.

Worked Example: One Buyer Question, One Window

Imagine a buyer asks an AI search engine, "What is the best contract management tool for a 50-person legal team?" Here is a simplified picture of what may happen inside the system.

The system reformulates the question into one or more searches. It retrieves perhaps ten candidate pages. It cannot hand all ten complete pages to the model, so it extracts passages, often a few hundred tokens each, that look most relevant. It adds its own instructions about tone, citation format, and safety. Then the model writes an answer of a few hundred tokens.

Now look at what decided the outcome. A vendor whose comparison page opens with "Built for in-house legal teams of 20 to 200 people, priced per seat" gives the retriever an obvious, self-contained match. A vendor whose page opens with an abstract mission statement and buries team size in a footnote may be retrieved but have the wrong passage selected. A vendor whose pricing exists only inside a PDF or behind a form may have no usable passage at all. None of these outcomes require the model to be biased. They come from what fit on the desk.

The exact pipeline differs by engine and is not fully published, so treat this as an illustration of the constraint rather than a description of any one product. The constraint itself, limited and selected context, applies broadly.

Common Misconceptions

"The model has read my whole website." Training data may include some of your public pages from a past crawl, but what the model learned in training is blurred and dated. For a live answer, the engine usually works from freshly retrieved fragments.

"A bigger context window means better answers." Only if the extra material is relevant and the model uses it well. More text can mean more noise.

"Word count equals token count." Tokens are usually more numerous than words, and the ratio shifts with language, jargon, numbers, and formatting.

"If it's in the window, the model will follow it." Instructions can conflict, get diluted, or be overridden by later text. Placement and clarity matter.

"Hidden reasoning is free." Where models bill reasoning tokens, they can be a major share of cost even when the visible answer is short.

A 30-Day Plan for B2B SaaS Teams

Week 1: Audit your core facts. List the ten facts buyers most need: what you do, who it is for, pricing model, integrations, security posture, and proof points. Check whether each one is stated plainly in a short passage somewhere on your public site.

Week 2: Rewrite for extraction. For each key page, make the opening two sentences answer the page's main question. Remove contradictions between pricing, product, and older content. Add dates to facts that change.

Week 3: Observe. Ask AI engines the questions your buyers ask, and record the prompt, engine, date, and answer verbatim. Note which passages appear to have been used and which facts are missing or wrong. Our guide to tracking LLM recommendations explains how to do this consistently.

Week 4: Fix and recheck. Address the gaps you observed, then rerun the same prompts. Treat changes cautiously, because answers vary between runs and engines change over time, but consistent improvement across repeated observations is a meaningful signal.

A Short Glossary

• Token: a piece of text, often part of a word, that a model reads or writes.
• Tokenizer: the component that splits text into tokens; it differs between model families.
• Context window: the maximum number of tokens a model can consider in one request, typically covering input and output together.
• Output limit: a separate cap on how many tokens a single response can contain.
• System prompt: instructions from the application builder placed at the top of the window.
• Truncation: dropping older content when the window fills.
• Retrieval: selecting relevant passages from documents or the web to place into the window.
• Reasoning tokens: internal tokens some models generate before the visible answer, often billed as output.
• Prompt caching: a provider feature that discounts or speeds up repeated identical input prefixes.
• Chunk: a section of a document, sized in tokens, that a retrieval system stores and returns as a unit.

Knowing these terms makes vendor documentation, pricing pages, and engineering conversations far easier to follow. When a provider advertises a large context window, you can now ask the useful follow-up questions: Does it include output? What is the output cap? How is reasoning billed? How well does the model use material placed in the middle of a long input?

The Bottom Line

Tokens are how a model counts text, and the context window is how much it can consider at once. Together they explain why AI tools forget, why costs vary, why long documents get uneven treatment, and why AI search engines describe your company from fragments rather than full pages. Teams that understand these limits write clearer prompts, build cheaper and safer AI features, and publish content that survives being read one paragraph at a time.

If you want to know which fragments AI engines are using to describe your brand today, and what they are getting wrong, book a strategy call with LLM Recommend.

Frequently Asked Questions

How many words is 1,000 tokens?

For ordinary English, roughly 750 words is a common estimate, but the ratio varies by tokenizer, language, jargon, numbers, and formatting. Measure with the specific model's tokenizer when accuracy matters.

Does the context window include the model's answer?

Usually yes. Instructions, history, attached material, tool results, and the answer in progress typically share the same budget, and many providers also cap output length separately.

Why does a chatbot forget what I said earlier?

The application resends history with each message. When the conversation exceeds the window, older turns are dropped or summarized, so earlier details may no longer be visible to the model.

Is a bigger context window always better?

No. It allows more material, but cost and latency rise, irrelevant text dilutes the useful passage, and research shows models can use information in the middle of long inputs less reliably than information at the start or end.

Why do API costs vary between similar requests?

Billing is per token, with output usually priced higher than input. Hidden reasoning tokens, resent conversation history, retrieved web content, and caching all change the real total.

What do context windows mean for my brand in AI search?

AI engines typically place selected passages, not your full site, into the model's window. Key facts should be stated plainly in short, dated, self-contained passages so the right fragment is selected and summarized accurately.