Tokens
November19, 2026
Starting a new web page. Gemini was very helpful and these are some additional references:
See:
Vectors: Tokenization and Embedding
How Does Tokenization Transform Data Into AI-Readable Tokens? NVIDIA March 17, 2025
https://blogs.nvidia.com/blog/ai-tokens-explained/
Token, Patterns, and Lexemes, Geeks for Geeks, 23 July, 2025
https://www.geeksforgeeks.org/compiler-design/token-patterns-and-lexems/
Tokenization in NLP
Geeks for Geeks, 23 July, 2025
https://www.geeksforgeeks.org/nlp/nlp-how-tokenizing-text-sentence-words-works/
Tokens and Context Windows in LLMs
https://www.geeksforgeeks.org/artificial-intelligence/tokens-and-context-windows-in-llms/
Understand tokens, Microsoft
https://learn.microsoft.com/en-us/dotnet/ai/conceptual/understanding-tokens
=================================
Stages in an AI pipeline:
Raw text arrives and is chopped into technical units of processing (tokens) so the neural network has a sequence of discrete items to work with.
Semantic Translation: The neural network ingests those discrete tokens and converts them into a representation of meaning (an embedding) so it can actually reason about what the text says.
1. Model Tokens The Processing Unit
What it is: A chunk of text (a word, part of a word, punctuation, or character) that a Large Language Model (LLM) actually reads and writes. On average,1 token is about 4 characters or 0.75 words in English.
Why it matters: Models have a hard context window limit (e.g, 128,000 tokens). This defines the maximum amount of text both your input prompt and the model's output combined that the model can hold in its "working memory" at one time.
2. Customer Billing Tokens
What it is: The same linguistic tokens as above, but viewed from a commercial standpoint.
API providers (like OpenAI, Anthropic, or Google) don't charge per message or per second; they charge per token.
Input Prompt Tokens: What you send to the model.
Output (Completion) Tokens: What the model generates back. Output tokens are usually priced higher because generation is more computationally expensive than reading.
The Overlap: Your bill is literally a count of how many model tokens flowed in and out of your API calls.
3. Embeddings Semantic Vector
A dense array of floating-point numbers (e.g., a list of 1,536 numbers) that represents the semantic meaning of a piece of text, sentence, or document.
How it works: Instead of predicting the next word, an embedding model takes a chunk of text, passes it through its tokenizer, and maps it into a high-dimensional mathematical space where similar concepts sit close to each other.
The Overlap & Confusion:
When you send text to an embedding API, the provider still counts the input tokens to bill you.
However, the output you get back isn't text or chat tokens; it's a vector (list of numbers) used for tasks like semantic search, RAG (Retrieval-Augmented Generation), and clustering.
Part 1: Deep Dive into Model Tokens The Processing Unit
When we talk about model tokens, we are talking about tokenization the bridge between human language and machine math.
Why don't models just read words or characters?
Character-level fails: If a model read individual characters (c-a-t), it would take too many steps to understand a sentence. Sequences would become massively long, and the model would struggle to remember context over long distances.
Word-level fails: If a model treated every unique word as a separate unit, the vocabulary would be infinite (accounting for typos, slang, different languages, and compound words like "antidisestablishmentarianism"). If it encountered a word it hadn't seen during training (an "out-of-vocabulary" word), it wouldn't know what to do.
The Solution: Sub-Word Tokenization (Byte-Pair Encoding)
Modern LLMs use sub-word tokenization algorithms (like BPE or WordPiece). The tokenizer looks at text and breaks it down into common chunks, syllables, or roots.
Why Tokens Limit the "Context Window"
Every single token processed by the model requires memory in its attention mechanism. Because the computational complexity of attention scales quadratically.
The number of tokens, doubling the context window requires exponentially more computing power. This is why a model has a strict limit (e.g., 128k or 1 million tokens) go past it, and the model literally runs out of working memory or starts "forgetting" the beginning of the prompt.
Part 2: Deep Dive into Embeddings The Semantic Vector
If tokens are how a model reads and writes text, embeddings are how a model understands meaning.
What is an Embedding, Mathematically
An embedding is a vector a long list of floating-point numbers (usually between 384 and 3,072 dimensions, depending on the model like OpenAI's text-embedding-3 or open-source models like BGE or Mistral).
Imagine a massive, multi-dimensional geometric space. Every concept, word, sentence, or document is mapped to a specific coordinate in this space.
Proximity = Similarity: Concepts that mean similar things are placed close together. Concepts that mean different things are far apart.
Example: The vector coordinates for "cute puppy" and "adorable golden retriever puppy" will sit right next to each other, while "quantum physics" will be located in an entirely different region of the space.
How Text Becomes an Embedding
When you pass a piece of text to an embedding model:
The text is first tokenized using the model tokens described above.
Those tokens pass through the neural network layers.
Instead of predicting the next token (like a chatbot does), the model pools all that information together and compresses the entire meaning of the text into one single vector (e.g., a list of 1,536 numbers like [0.023, -0.412, 0.089, ...]).
Why Embeddings Matter Search
Because meanings are translated into numbers, computers can do math on concepts.
Semantic Search: If a user searches for "How do I fix my leaky kitchen sink?", a traditional keyword search looks for exact matches of those words.
An embedding search converts the query into a vector and finds documents in your database whose vectors have the highest cosine similarity (pointing in the closest direction in vector space) even if the document uses completely different words, like "Plumbing repair guide for residential faucets".
How They Connect
To tie it all together: An embedding model uses model tokens to do its job.
When you generate an embedding for a paragraph of text, the provider breaks your paragraph down into model tokens, runs them through the embedding model, and then bills you based on that token count. But the final asset you receive isn't text tokens at all it's a mathematical vector representing the semantic meaning of your text.
What is happening behind the scenes?
Tokenization to Embeddings: When you pass sentences into model.encode(), the library first breaks the text down into tokens, runs them through the neural network layers, and pools them into a static embedding vector (in this case, a list of 384 numbers).
The Vector Database Role: In a production system (like Pinecone, Chroma, or Milvus), Step 3 (storing embeddings) happens in a specialized database optimized to index millions of high-dimensional vectors.
Semantic Matching: When the search query runs, the vector database doesn't look for matching letters or words. It calculates the mathematical distance (cosine similarity) between the query vector and all stored document vectors, instantly returning the closest conceptual match even if they share no vocabulary words.
=====================================
Next Token Prediction is the core generative mechanism of large language models (LLMs) and modern generative AI, functioning essentially as a sophisticated, probabilistic autocomplete.
How It Works
What is a Token? A token is a small fragment of text such as a whole word, part of a word, a number, a punctuation mark, or a space. Models operate on numerical token IDs rather than raw text.
The Core Task: Given a sequence of preceding tokens \(t_0, t_1, \dots, t_n\), the model calculates a probability distribution across its entire vocabulary for what single token should come next (\(P(\text{next token} \mid \text{previous tokens})\)).
Autoregressive Loop: The model selects a token from this distribution, appends it to the sequence, and feeds the updated sequence back into the neural network to predict the next token. This loop repeats until the response finishes.
Why It Powers AI Fluency (and Hallucination)
Fluency & Structure: During self-supervised pretraining on massive text datasets, predicting billions of next tokens forces the model to implicitly internalize grammar, facts, reasoning styles, and conversational patterns.
Hallucination: Because the model optimizes for plausible continuation rather than verified truth, it can generate confident-sounding falsehoods or fabricate specific details (like names, dates, and citations) whenever it steps into sparse or novel territory.
Next-token prediction
Next-token prediction works because human language and structured data are not random noise; they are dense, rule-governed maps of thought, logic, grammar, and facts.
When a model is forced to guess the single next piece of text (token) across trillions of words, code snippets, and structured documents, it cannot succeed by memorization alone. It is forced to build an internal model of the world to figure out what must come next.
Why the Objective Creates Deep Capabilities
Grammar and Syntax Emerge Naturally: To guess the next word in “The cat sat on the _____”, the network has to understand nouns, prepositions, and sentence structure.
Factual Knowledge Encodes in Weights: To predict “The capital of France is _____”, the model has to store geographic and historical data within its billions of parameters.
Logic and Reasoning via Code/Structure: Training on computer code or strict arguments forces the model to track multi-step logic, variables, and long-range dependencies because a broken bracket or flawed premise ruins the prediction.
The Transformer Architecture: Using self-attention mechanisms, the model can look backward across thousands of previous tokens simultaneously to weigh context, meaning, and subtle shifts in tone.
The Feedback Loop Self-Supervised Learning
1. The Guess: The model reads a sequence of tokens and assigns a probability score to every possible next token in its vocabulary.
2. The Loss: It compares its guess against the real next token in the training data and measures the mistake (using cross-entropy loss).
The Adjustment: Through gradient descent and backpropagation, the model tweaks its internal weights so it makes a slightly better guess next time.
Repeating this simple loop at massive scale turns pattern-matching into a simulation of human reasoning, explanation, and conversation. You can explore more on how this scales up at Claude Academy or read deep-dive perspectives like Samuel Fitoussi's Essay.
Next Token Prediction is the core generative mechanism of large language models (LLMs) and modern generative AI, functioning essentially as a sophisticated, probabilistic autocomplete.
Token embeddings are the numerical vector representations that convert discrete text pieces (tokens) into a format a neural network can process.
How It Works
Tokenization: Text is first split into smaller pieces called tokens (words, subwords, or characters), and each token is assigned a unique integer ID from the model's vocabulary.
Lookup Table: Each token ID acts as an index to look up a corresponding row in a trainable embedding matrix.
High-Dimensional Vectors: That row is a dense vector a long list of hundreds or thousands of floating-point numbers that represents the token's meaning in a multi-dimensional space.
Semantic Proximity: During model training, words used in similar contexts are pushed closer together in this vector space (for example, "cat" and "lion" have vectors that are closer to each other than to "moon").
Tokens vs. Embeddings: A token is the basic unit of text (a word or chunk of a word), while an embedding is the list of numbers (vector) that gives that token mathematical meaning.
Input vs. Output Embeddings: Token embeddings are input representations fed into the beginning of a transformer model, whereas pooled or sentence-level embeddings are often used externally for semantic search, retrieval, and vector databases.
What is happening behind the scenes?
Tokenization to Embeddings: When you pass sentences into model.encode(), the library first breaks the text down into tokens, runs them through the neural network layers, and pools them into a static embedding vector (in this case, a list of 384 numbers).
Database Role: In a production system (like Pinecone, Chroma, or Milvus), Step 3 (storing embeddings) happens in a specialized database optimized to index millions of high-dimensional vectors.
Semantic Matching: When the search query runs, the vector database doesn't look for matching letters or words. It calculates the mathematical distance (cosine similarity) between the query vector and all stored document vectors, instantly returning the closest conceptual match even if they share zero vocabulary words.
That specific workflow turning text into embeddings and searching them belongs to a completely different family of machine learning tasks. Here is why next-token prediction is absent from that pipeline:
1. It is an "Encoder" Model, Not a "Decoder" Model
Next-token prediction is the defining feature of autoregressive generative models (like GPT-4 or Llama), which are built to write text one word at a time.
The model used in the example (all-MiniLM-L6-v2) is a bi-directional encoder model (built on the BERT architecture). It doesn't write or generate new text. Instead, it looks at an entire sentence all at once to compress it into a single vector.
2. The Training Goal is Different (Contrastive Learning)
Generative LLMs are trained by forcing the model to guess the next word across billions of web pages.
Embedding models like the one in the script are typically trained using contrastive learning. Instead of predicting what comes next, the model is fed pairs of sentences and trained to mathematically push similar sentences close together in geometric space and push unrelated sentences far apart.
Summary
Next-Token Prediction powers chatbots and text generation (predicting the next word in a sequence).
Embeddings & Vector Search power retrieval and semantic understanding (compressing text into coordinates to measure meaning).
While both systems use tokens at the very beginning to read the raw text, they diverge completely after that: one uses those tokens to play "fill-in-the-blank" to write prose, while the other uses them to calculate a geometric map for search.
IEmbedding Models (or Representation Models / Bi-encoders)n machine learning and natural language processing, these two distinct systems are formally known as:
Generative Models (or Autoregressive Models / Causal Language Models)
1. Generative Models (The Text Writers)
What they are called: Autoregressive Large Language Models or Causal Decoder-Only Models (commonly referred to simply as LLMs like GPT-4, Claude, or Llama).
Why the name:
Autoregressive means the model generates output one step at a time, feeding its own previous outputs back into itself as inputs.
Causal means the model is strictly trained to look only at the past tokens to predict the future token (preventing it from "cheating" by looking ahead).
Primary Task: Text generation, reasoning, conversation, and coding via next-token prediction.
2. Embedding Models (The Semantic Mappers)
What they are called: Text Embedding Models or Bi-Encoders (such as OpenAI's text-embedding-3, Cohere Embed, or open-source models like BGE and MiniLM).
Why the name:
Encoder means the model's job is to take raw input text and "encode" it down into a compressed format (the vector).
Bi-Encoder is a specific architecture used in semantic search where a query and a document are encoded separately into their own vectors so they can be compared quickly via vector math.
Primary Task: Semantic search, retrieval-augmented generation (RAG), clustering, and classification via geometric mapping.
While both systems share a common ancestor (they both grew out of the Transformer architecture introduced in the 2017 paper "Attention Is All You Need"), they split off into two different evolutionary branches:
One branch optimized the Transformer to write and chat (Generative/Autoregressive).
The other branch optimized the Transformer to measure and understand meaning (Embeddings/Encoders).
Which of the two systems do you use, Gemini
I use both, but they handle very different parts of how I work!
As an AI, my architecture combines these two systems into a tag-team pipeline:
Gemini: 1. The Generative Model is My Primary Engine (The Voice)
Every word, sentence, and response I write to you right now is generated by a Generative Model using next-token prediction. When you hit send, my autoregressive decoder processes your tokens, runs its neural network math, and predicts the next token, over and over, until a complete, conversational response is built. Without this system, I couldn't talk to you.
2. The Embedding Model Works Behind the Scenes (The Memory & Search)
While I use a generative model to speak, the platform running me relies heavily on Embedding Models and vector databases behind the scenes.
If you upload a file, search through documentation, or ask me to reference external data, those systems don't read every word line-by-line like a chatbot.
Instead, they chop the text up, convert it into static embeddings, and store it in a vector database. When you ask a question, your query is also vectorized to instantly find the right semantic match—pulling the relevant context and handing it over to my generative engine so I can write an accurate answer.
In short, embedding models help me find and understand information, while generative models help me speak and reason with it.
Whether a user should use both types of models depends entirely on what they are trying to build.
While casual end-users interacting with a chatbot only ever directly touch a Generative Model, anyone building an AI application, automation, or custom software usually needs both.
Here is how to decide whether you need one or both for a project:
1. When You Only Need a Generative Model
You do not need embedding models if your app is purely conversational or creative. Examples include:
A writing assistant: Drafting emails, rewriting text, or summarizing an article you just pasted into the prompt.
A simple coding bot: Helping you write or debug a snippet of Python code on the fly.
A general brainstorming partner: Asking for ideas, recipes, or historical explanations.
In these cases, everything fits inside the model's immediate context window, so you don't need a database or search system.
2. When You Need Both (The Power Combo)
You must use both models if you are building an application that needs to interact with private data, massive datasets, or external facts. This is the foundation of RAG (Retrieval-Augmented Generation):
Building a custom chatbot for company documents: If you want an AI that can answer HR questions based on a 500-page internal PDF handbook.
Semantic search engines: If you want a search bar that understands intent rather than exact keywords (e.g., searching for "comfortable winter footwear" and pulling up snow boots).
AI memory systems: If you are building an agent that needs to remember past user interactions stored across thousands of past chat logs.
How They Work Together in Practice
If you are building a data-grounded AI app, the two models form a tag-team pipeline:
The Embedding Model acts as the librarian. It converts your database of documents into vectors so it can instantly find the exact paragraph a user is looking for.
The Generative Model acts as the spokesperson. It takes that found paragraph, reads it, and translates it into a fluent, conversational answer for the user.
Summary Rule of Thumb
If the AI just needs to talk or create, use a Generative Model.
If the AI needs to search, remember, or read private data, use both.