Static to Contextual Embeddings

Tokenization turns text into stable token IDs. The harder question is what those IDs should mean once a model starts learning from them.

A tokenizer can turn short sentences into IDs:

IDs solve naming. They don't tell a model that encoder resembles decoder, or that charge means an invoice line in one sentence and supplying power in another.

An embedding gives each token a vector of numbers. A contextual representation goes one step further: it adjusts that vector using the sentence around this particular occurrence.

You'll build both ideas from scratch and watch the same token leave with two different meanings.

Look up coordinates for token IDs

Suppose a tokenizer has a vocabulary of size V, and you choose embedding dimension d. The model stores an embedding matrix E with shape V x d.

• Row E[token_id] is the starting vector for that token.
• Training changes rows so tokens used in similar situations become useful to the prediction task.
• Before context mixing, every occurrence of the same token ID receives the same row.

That final point matters. If charge is ID 0, both billing and device sentences initially retrieve row E[0].

A lookup table alone can't distinguish senses. First, though, it has to learn a useful map of token usage.

Learn meaning from neighboring words

Words that repeatedly appear near similar words tend to play similar roles. In technical text, encoder and decoder may both occur near layer, attention, or mask; cache and prefix may cluster around reuse and latency language.

The simplest evidence is a co-occurrence count: for every token, count words within a fixed context window.

A count row can be high-dimensional and noisy. One historical route, latent semantic analysis, uses singular value decomposition (SVD) to compress a high-dimensional term-document matrix.^{[1]Reference 1Indexing by Latent Semantic Analysishttps://doi.org/10.1002/(SICI)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9} Here, use the same compression move on word-context counts. The compressed rows act as static vectors: one vector per word, independent of sentence.

The next exercise follows that word-context route. It computes positive pointwise mutual information (PPMI), which emphasizes pairs occurring more often than chance, then compresses the matrix with SVD.

This tiny dataset makes the pattern visible. Real corpora have far more contexts, imperfect wording, and competing meanings.

Predict neighbors with Word2Vec

Counting first and compressing later isn't the only way to learn a static embedding table. Word2Vec trains vectors through prediction:

• CBOW predicts a center token from its surrounding tokens.
• Skip-gram predicts surrounding tokens from a center token.

Both objectives reward vectors that help predict observed neighborhoods. Mikolov and colleagues introduced these two architectures in 2013.^{[2]Reference 2Efficient Estimation of Word Representations in Vector Space.https://arxiv.org/abs/1301.3781}

For Skip-gram, a training set can be built directly from windows. With window size 2, the center arrived produces one positive pair for each visible neighbor.

Predicting every vocabulary item for every pair is costly. Negative sampling trains the observed pair as positive and a small set of sampled unobserved pairs as negative. This objective is a small, inspectable version of that idea.

No analogy trick is required to understand the useful result: after enough examples, tokens that support similar neighbor predictions can end up near one another.

Fit global counts with GloVe

GloVe starts from global co-occurrence counts. It fits a weighted least-squares objective: a word-vector and context-vector dot product, plus biases, should approximate the logarithm of each observed count. The logarithm compresses the count range, while the weighting function limits how much very common pairs dominate training.^{[3]Reference 3GloVe: Global Vectors for Word Representation.https://aclanthology.org/D14-1162.pdf}

Word2Vec and GloVe give each known word one static row. That's efficient, but it creates two problems: new spellings have no row, and ambiguous words have only one.

Compose new spellings from pieces

Technical corpora constantly encounter variants: tokenize, tokenized, tokenizer, or misspellings mixed into text. FastText represents a known word using character n-grams as well as its whole-word identity, letting related spellings share parameters.^{[4]Reference 4Enriching Word Vectors with Subword Information.https://arxiv.org/abs/1607.04606} For an out-of-vocabulary word, there's no learned whole-word row. FastText can still sum its learned n-gram vectors into a usable representation.

The code checks which pieces overlap. A trained FastText model would sum learned vectors for those pieces and omit the missing whole-word row for an out-of-vocabulary spelling.

Subword composition helps with unfamiliar surface forms. It still doesn't decide which meaning an existing ambiguous token carries.

One static row can't choose a sense

Read these messages:

The word charge is spelled the same way in both. A static embedding lookup returns the same vector, even though the first case belongs near billing language and the second near device language.

Because the two charge vectors are identical, every downstream comparison begins from the same mistaken compromise. The representation needs evidence from the sentence.

Let tokens exchange context

ELMo showed that a token's representation can be produced from a bidirectional language model and selected from multiple learned layers, rather than stored as one context-free vector.^{[5]Reference 5Deep contextualized word representations.https://arxiv.org/abs/1802.05365}

BERT later trained a Transformer encoder with masked language modeling: hide some tokens, then predict them using context on both sides. For a visible token state, encoder self-attention can incorporate evidence before and after that token.^{[6]Reference 6BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}

You don't need a pretrained model to see the central mechanism. In this toy contextualizer, the same starting vector for charge is mixed with a clue from its sentence.

The weights above are illustrative, not learned model parameters. A trained attention layer learns which context tokens should contribute, and later layers can refine that state again.

Bidirectional and causal context

Not every contextual representation may look to the same places.

• A BERT-style encoder token can use tokens on both sides of its position during encoding.
• A GPT-style causal language model predicts the next token from the prefix, so a token state can't use later tokens while preserving that next-token objective. The original GPT work used a Transformer decoder for generative pretraining.^{[7]Reference 7Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}

For the first token in charge the scanner, the right-hand clue scanner is visible to an encoder but not to a causal decoder state at position charge.

Both designs produce contextual states. Their visibility rules suit different training objectives and later tasks.

Similarity needs a geometry check

Embedding applications often use cosine similarity: a value near 1 means two vectors point in similar directions. That measurement is useful only if the space behaves sensibly.

Research on contextual representations found that token vectors can be anisotropic: many vectors share a dominant direction, inflating raw cosine similarities even for tokens that shouldn't be treated as alike.^{[8]Reference 8How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.https://arxiv.org/abs/1909.00512}

The two states share a large horizontal component. Raw cosine says they're almost identical; subtracting their shared mean reveals that their informative vertical components oppose one another.

Centering isn't a universal production recipe. It's a diagnostic reminder: measure retrieval or classification quality on held-out examples instead of trusting a similarity score in isolation.

Choose what the failure requires

No representation is best by slogan. Start with the failure mode and the measurement that would prove improvement.

Contextual token states aren't automatically good message-level retrieval vectors. Pooling, task training, and evaluation still matter. You'll build those choices in later retrieval lessons.

From lookup rows to occurrence states

The chain from token IDs to context-dependent meaning is now concrete:

1. An embedding table maps each token ID to one starting vector.
2. Co-occurrence, Word2Vec, and GloVe learn static geometry from usage evidence.
3. FastText-style character pieces let related spellings share parameters.
4. Static vectors fail when identical token text carries different senses.
5. Context mixing gives each occurrence its own state.
6. Cosine similarity still requires evaluation because geometry can be distorted.

Mastery check

What strong answers show

• Foundational: You can explain why an embedding lookup returns one stored row for every occurrence of a token ID.
• Intermediate: You can build a co-occurrence or prediction example and show why the two meanings of charge require context mixing.
• Advanced: You can choose a representation for a measured failure case and test whether cosine geometry supports the chosen metric.

Follow-up questions

When embeddings mislead

• Treating token IDs as meaning: An ID only selects a row; training and context create useful geometry.
• Calling subword handling disambiguation: Character pieces help unfamiliar spellings, not multiple senses of one familiar word.
• Trusting cosine without an evaluation set: A high score can come from shared directions rather than task-relevant similarity.