Tokens and AI
Every answer starts with small pieces. See how words become tokens, how context shapes a reply, and why your judgment matters.
AI chat tools can draft a summary, rewrite an email, or answer a policy question in seconds. Fluent language can create a strong impression of understanding.
These tools are a kind of generative AI: AI that creates content from learned patterns, such as text, images, audio, video, or code. A draft message and a summary are both generated content.
The model, the AI system behind the tool, builds a response one small piece at a time. Each piece is called a token. The model gives possible next tokens different chances. The tool picks one, based on its settings, and repeats. Several tokens may fit, which helps explain why the same prompt can produce different wording.
Your words become tokens
The model breaks text into small pieces called tokens. A token can be a whole word, part of a word, a space, or punctuation.
Pick a sentence and see how it splits into tokens.
These are real splits from a widely used AI model. A space symbol marks the space that most pieces carry before the word. Other AI systems split text their own way.
Each token ID points to a long list of numbers inside the model, often thousands of them. The model does its work on those numbers.
Tokens are also how AI use is measured
AI services count the tokens in what you send and in the reply they write. Some charge by the token, and many use token counts to set usage limits and how much text one conversation can hold.
Pasting a long document, keeping a very long chat going, and asking for a long answer all use more tokens. In a chat, each new message is sent along with the conversation so far, so the count grows as the chat does.
Go deeper: how the model learned these patterns
Related tokens end up near each other
During training, tokens used in similar ways end up with similar lists of numbers. Drawn on a flat map, related tokens form neighborhoods. This drawing is a teaching picture: real lists have thousands of directions, far more than two.
Training nudges many internal numbers
↻ Compare with the real text, then nudge
During training, the model practices guessing what text comes next. The training process compares each guess with the real text, then nudges many internal numbers a little. Repeating this across a huge amount of text builds the patterns.
Then it learns to follow instructions
Chat tools get a second round of training. People write example answers to instructions for the model to learn from, then rank its responses so it learns which ones people prefer. One common method is called reinforcement learning from human feedback.
This step shapes the tone and helpfulness of the answers. The model still writes by predicting the next token, and training methods differ across AI systems.
Context changes what a token means
The model uses the words that come before a token as clues. The same token can play a different role in a different sentence.
Simplified teaching picture. Real models weigh many earlier words at once.
“Charge” in a billing sentence
Billing added the charge to the account.
The earlier words “billing” and “added” point to a fee on an account.
“Charge” in a staffing sentence
The nurse in charge updated the schedule.
The earlier words “nurse” and “in” point to the person leading the shift.
One token, two uses
Billing asked the nurse in charge to review the charge.
Both uses of “charge” get the same token ID. The words before each one steer how the model uses it. Changing your wording can change the answer you receive.
A reply grows one token at a time
The model gives each possible next token a chance. The tool picks one, adds it to the text, and repeats.
Try it. You can see the patient's chart. The model receives the text you put in the prompt. Build the sentence yourself, then let the model run the same prompt a few times and compare each run with the chart. Then add context to the prompt and run it again.
Finish a sentence to log it here.
Underlined: where this run first split from the one before it.
Teaching example: these chances are made up to show the idea. A real model gives a score to every token in its vocabulary at each step, often 100,000 or more.
Simplified: each choice here is a whole word, and this lab looks only at the last word and whether the note is included. A real model uses all the text so far, and some words take more than one token. Many tools set the temperature for you, and some let you adjust it.
Temperature is the tool's setting. The prompt is yours.
In the lab, the sentence start alone never said where the patient went or why, so the model filled in those details. Adding the note so far moved the likely wording toward the chart. A specific prompt goes further and leaves even less to fill in.
Write a note about a patient who was admitted.
Leaves the unit, the reason, and every other detail for the model to invent.
Using only this chart entry, write one sentence about the admission. If something isn't in the entry, say it isn't documented. Entry: Admitted to the ICU with sepsis.
Supplies the facts, limits the answer to them, and gives the tool a way to say what's missing.
Specific prompts reduce made-up details. Some can still slip through, so check the result against the source.
Why a fluent answer can still be wrong
AI can produce a sentence that sounds convincing even when a claim is false or unsupported. This is often called a hallucination.
“According to a 2021 study in the New England Journal of Medicine (Hartman et al.), the protocol reduced readmissions by 34%.”
This citation has no supporting source. Its author, journal, year, and statistic fit the pattern of academic writing. Checking against trusted evidence is what establishes support for a generated claim.
You saw the same thing in the lab: every run read like a chart note, and only the chart settled which one was true.
Fluency, speed, and confidence provide no guarantee of accuracy. A tool can state a fabricated drug dose, policy number, or patient instruction in the same calm tone as a supported claim.
Your judgment is one part of the safety system. Use approved tools, verify important claims against trusted sources, follow required review and escalation steps, and remain accountable for what you send, sign, or act on.
What this means for your workday
Use it for first drafts
Summaries, emails, rewrites, brainstorming. It can give you a useful first draft quickly. You decide what stays.
Always verify facts
Names, numbers, dates, citations, dosages, policies. If it matters, check it against a trusted source before you rely on it.
Keep private data in approved tools
Public AI tools include free, personal, and organizationally unapproved services. Keep patient information and confidential data inside the approved, secured tools your organization provides.
You're accountable for the output
The AI drafts and you are the author. Whatever you send, sign, or act on is your responsibility, the same as always.
Three quick scenarios
Pick an answer to see feedback. Nothing is scored or saved.
Sources
The lesson stays in plain language. Technical references are available here for readers who want them.
View technical sources and further reading
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). 2024. doi.org/10.6028/NIST.AI.600-1 Source for the generative AI definition used on this page and in the G4E glossary.
- OpenAI. tiktoken (o200k_base encoding), version 0.14.0. Splits on this page generated September 26, 2026. github.com/openai/tiktoken Shows how text is divided into tokens, including whole words, parts of words, spaces, and punctuation.
- Vaswani A, et al. Attention Is All You Need. 2017. arxiv.org/abs/1706.03762 Introduces the design behind many language models, including the rule that, while generating text, each position draws only on earlier positions.
- Ethayarajh K. How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. 2019. aclanthology.org/D19-1006 Shows that the same word can be represented differently in different contexts.
- Llama Team, Meta. The Llama 3 Herd of Models. 2024. arxiv.org/abs/2407.21783 Reports a vocabulary of 128,000 tokens and thousands of numbers per token inside the model.
- Ouyang L, et al. Training language models to follow instructions with human feedback. 2022. arxiv.org/abs/2203.02155 Describes the second round of training: people write example answers, then rank the model's responses so it learns which ones they prefer.
- He H, Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. 2025. thinkingmachines.ai Explains why the same prompt can return different answers, even with settings meant to make output repeatable.
- OpenAI. API reference: Create chat completion, temperature. Accessed September 26, 2026. developers.openai.com Describes a setting that makes answers more focused or more varied.
- Anthropic. Messages API reference, temperature. Accessed September 26, 2026. platform.claude.com Notes that even the lowest temperature setting does not make results fully deterministic.
- Anthropic. Pricing. Accessed September 27, 2026. platform.claude.com Shows usage priced per million tokens, with the tokens you send and the tokens the model writes priced separately.
- Anthropic. Context windows. Accessed September 27, 2026. platform.claude.com Explains that the conversation history, the new message, and the reply all count toward the token limit, and that each turn carries the earlier turns forward.
- Anthropic. Reduce hallucinations. Accessed September 27, 2026. platform.claude.com Recommends grounding answers in supplied documents, limiting the model to that information, and letting it say when it doesn't know. Notes these steps reduce errors without removing them.
- World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: WHO; 2024. who.int/publications/i/item/9789240084759 Documents the risk of false or unsupported information produced by large multi-modal models, including large language models, in health.