Static to Contextual Embeddings

The previous lesson turned text into token IDs. Those IDs are addresses, not meanings. The same ID can appear in a billing dispute and in a device instruction:

Both lines contain ID 407, yet invoice and scanner point to different uses. If both lookups start from one row, where can that difference enter? An embedding stores a vector of numbers for each ID. A contextual representation can then rewrite that vector using words around this particular occurrence.

First inspect the static table. Then watch the same token leave with two different states.

The first stop is the lookup. Its limitation gives context mixing a job.

A token ID selects one stored row

Let vocabulary size be V and embedding dimension be d. The model stores an embedding matrix E with shape V x d.

The table answers one narrow question: given an ID, which vector enters the model?

• Row E[token_id] is the starting vector for that token.
• Training moves rows so tokens used in similar situations become useful for the prediction task.
• Before any context mixing, every occurrence of the same ID receives that same row.

A one-hot row vector of length V with a 1 at token_id selects the same row. Multiplying it by E writes the lookup as matrix algebra.

If charge is ID 0 in a tiny table, both billing and device sentences start at E[0]. A production tokenizer might use 407 instead. The rule doesn't change: the integer is an index, not a sense label.

The lab uses a three-row table so every index is visible. Before running it, predict whether the two printed rows can differ.

The rows are equal, so no lookup-only operation can resolve the two senses. Training can make one row useful across its uses, but only a context-dependent computation can specialize an occurrence.

Nearby words leave a fingerprint

If encoder and decoder keep seeing layer and mask, their neighborhood fingerprints start to resemble each other. The same logic puts cache near prefix when both occur with reuse language. This is the distributional hunch: usage supplies a weak signal about role.

The simplest measurement is a co-occurrence count. For each token, count words inside a fixed context window.

The sample exposes the fingerprints: encoder and decoder share layer and uses, while cache sees prefix and reuse language. The rows are still counts, not compact coordinates.

A count row is high-dimensional and noisy. Latent semantic analysis compressed a term-document matrix with singular value decomposition (SVD).^{[1]Reference 1Indexing by Latent Semantic Analysishttps://doi.org/10.1002/(SICI)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9}

The same move works on word-context counts: keep directions that explain the main neighborhoods, drop the rest.

Raw counts can mistake a common word for an informative neighbor. Positive pointwise mutual information (PPMI) keeps pairs that occur more often than chance and floors negative values at zero.

If #(w, c) is the count for word w with context c,

$$\text{PMI}(w, c) = \log \frac{\#(w, c) \cdot N}{\#(w)\, \#(c)}, \qquad \text{PPMI}(w, c) = \max(\text{PMI}(w, c), 0)$$

where N is the total of all counts, #(w) is the row sum, and #(c) is the column sum.

After compression, cosine similarity asks whether two vectors point the same way, ignoring length:

$$\cos(u, v) = \frac{u \cdot v}{\|u\|\, \|v\|}$$

Parallel vectors score 1. Orthogonal vectors score 0. Opposite vectors score -1.

The next lab makes the geometry visible. It starts with a 4×4 fixture: architecture words (encoder, decoder) share layer and mask; serving words (cache, prefix) share hit and reuse. It converts counts to PPMI, then keeps two SVD components. The power iteration is a small stand-in for a full SVD: find the strongest direction, subtract it, then find the next one.

Predict the result before running it: which pair should have cosine 1, and which pair should be separated?

The architecture rows become parallel. The serving rows land on a second axis at right angles to the first. Similar neighborhoods became similar directions, which is the distributional hunch made geometric.

The toy blocks make the split obvious. Real corpora add messy wording, rare contexts, and words that play several roles. Counting is one route to this geometry, not the only route.

Predict the neighbors instead

Counting is explicit. Word2Vec learns the same kind of static table by trying to predict neighbors:

• CBOW (continuous bag-of-words) predicts a center token from its surrounding tokens.
• Skip-gram predicts surrounding tokens from a center token.

There is no label saying “similar.” Each correct prediction nudges vectors so tokens that support similar predictions can share a neighborhood. Mikolov and colleagues introduced the two architectures in 2013.^{[2]Reference 2Efficient Estimation of Word Representations in Vector Space.https://arxiv.org/abs/1301.3781}

Skip-gram turns each window into training pairs. With window size 2, center hit produces one positive pair for each visible neighbor. The center stays first in each pair.

Count those pairs before looking at the output: the window reaches request, cache, and layer, but not today.

The output confirms the boundary: today is three positions away, so it never becomes a positive pair for hit. Each listed pair supplies one local prediction target.

Scoring every vocabulary item for every pair is expensive. A Skip-gram trainer can instead treat the observed neighbor as a positive example and draw a few unobserved tokens as negative samples. Levy and Goldberg showed that Skip-gram with negative sampling is closely related to factorizing a shifted PMI matrix. That link explains why prediction geometry often agrees with the count-based SVD geometry above.^{[3]Reference 3Neural Word Embedding as Implicit Matrix Factorization.https://arxiv.org/abs/1405.4053}

The loss below is a small, inspectable version of that update. It pushes the center toward the true neighbor and away from sampled ones. σ is the logistic sigmoid.

$$-\log \sigma(u \cdot v_+) - \sum_{n \in \mathcal{N}} \log \sigma(-u \cdot v_n)$$

A high positive dot product lowers the first term. A negative dot product lowers each negative-sample term.

The observed pair has lower loss, so training prefers it. Repeating that update puts tokens with similar neighbor predictions near one another. No analogy puzzle is required to see the signal.

Local windows are one view. GloVe starts with the global table of how often each pair occurred anywhere in the corpus.

Fit the global count table

GloVe starts from those global co-occurrence counts. It fits a weighted least-squares objective: a word vector and a context vector, plus two biases, should reconstruct the logarithm of each observed count.^{[4]Reference 4GloVe: Global Vectors for Word Representation.https://aclanthology.org/D14-1162.pdf}

The dot product is the model's compatibility score for that word-context pair.

$$w_i \cdot \tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}$$

The logarithm compresses counts that span many orders of magnitude. A weighting function then limits how much very common pairs dominate:

$$f(x) = \begin{cases} (x / x_{\max})^{\alpha} & \text{if } x < x_{\max} \\ 1 & \text{otherwise} \end{cases}$$

The paper's default is $x_{\max} = 100$ and $\alpha = 3/4$. Rare pairs get small weights. Very frequent pairs stop gaining extra weight past the cutoff.

The lab fixes the model score at log(30). Predict which count has zero residual, then see how weighting changes with frequency.

The score matches log(30), so count 30 has zero residual. Count 3 misses that target and pays a residual of 5.302; the weighting function also gives rarer pairs less influence.

Word2Vec and GloVe still give each known spelling one static row. That makes lookup cheap, but leaves two gaps: a new spelling has no row, and an ambiguous word has only one.

Build unknown spellings from pieces

Technical text constantly produces variants: tokenize, tokenized, tokenizer, plus typos mixed into logs. FastText represents a known word with character n-grams as well as its whole-word identity, so related spellings share parameters.^{[5]Reference 5Enriching Word Vectors with Subword Information.https://arxiv.org/abs/1607.04606}

The paper uses n-gram lengths 3 through 6 with boundary markers. The lab uses length 3 so every window is visible.

For an out-of-vocabulary (OOV) spelling, there's no learned whole-word row. FastText can still sum n-gram vectors it has already learned.

The output isolates the reusable pieces. A trained FastText model would sum learned vectors for them. For tokenized, it would skip a whole-word row that was never stored.

Subword composition helps with unfamiliar surface forms. It still can't decide which meaning an already-known ambiguous token carries.

One static row can't choose a sense

Read these two messages:

The spelling charge is the same. A static lookup returns the same vector, even though the first case belongs near billing language and the second near device language.

Before running the lab, predict whether either similarity score can notice that difference.

Both comparisons match because the two charge vectors are identical. Every downstream decision therefore starts from the same compromise. Sentence evidence has to enter after the lookup.

Mix in a neighbor from this sentence

The lookup stays fixed, so context enters in layers above it. A Transformer can learn a weighted mix of neighboring states.

To isolate that move, take the shared charge row and average it with one clue from its sentence: billing with invoice, device with scanner.

The two starts are identical. Predict where the final states should move before running the lab.

The equal weights are illustrative, not learned parameters. Billing charge moves toward invoice (cosine 0.879), while device charge stays farther away (0.355). A trained attention layer learns which neighbors should contribute, and later layers can refine the state again.

ELMo made this context-dependent representation explicit. A bidirectional language model produces token states, then a downstream task learns a weighted mix of its layers instead of using one context-free row.^{[6]Reference 6Deep contextualized word representations.https://arxiv.org/abs/1802.05365}

At position k, with layer states $h_{k,0}$ (character CNN) through $h_{k,2}$ (the top bidirectional LSTM),

$$\text{ELMo}_k = \gamma \sum_{j=0}^{2} s_j h_{k,j}$$

The task-specific weights $s_j$ are normalized so they sum to 1. $\gamma$ scales the whole mix. Lower and upper layers can therefore contribute different kinds of information.

BERT changed how deep bidirectional context is trained. Its masked language-model objective hides selected tokens and predicts them from both sides, so encoder self-attention can read evidence before and after a position.^{[7]Reference 7BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}

The binary visibility mask says who may attend. Learned attention weights say how much each allowed token contributes. Those are separate objects.

Not every model may look both ways

A BERT-style encoder token can use tokens on both sides during encoding. A GPT-style causal language model predicts the next token from the prefix, so its state at a position can't use later tokens while preserving that objective.

The original GPT work used a Transformer decoder for generative pretraining.^{[8]Reference 8Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}

At the first token in charge the scanner, the right-hand clue scanner is visible to an encoder and hidden from a causal decoder state at charge. Predict which list each mask should produce.

Both designs produce contextual states. Their visibility rules match different training objectives: one can use the whole input, while the other must preserve prefix-only prediction.

Even when two states are allowed to differ, a similarity score can still lie to you.

A shared direction can fake similarity

Embedding applications often use cosine similarity: a value near 1 means two vectors point in similar directions. That score can mislead when the space shares a large, uninformative direction.

Work on contextual representations found that token vectors can be anisotropic: many vectors share a dominant direction, so raw cosine stays high even for tokens that shouldn't be treated as alike.^{[9]Reference 9How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.https://arxiv.org/abs/1909.00512}

Two states can disagree in a task-relevant component while still pointing almost the same way overall.

Predict what raw cosine will say before running the lab. The two states share a large horizontal component, so raw cosine treats them as almost identical. Subtracting their shared mean exposes vertical components that point in opposite directions.

The output makes the warning concrete: raw cosine is 0.98, while centered cosine is -1.0. Centering isn't a universal production recipe. Use it as a diagnostic, then measure retrieval or classification quality on held-out examples instead of trusting a similarity score in isolation.

Match the representation to the failure

Choose a representation from the failure mode and the measurement that could prove an improvement. No representation wins by name alone.

Contextual token states aren't automatically good message-level retrieval vectors. Pooling and task training change what a vector means, so evaluate those choices on the retrieval task.

Later retrieval lessons pick them up.

The same geometry feeds next-token prediction. A model scores candidate tokens from the current state, then turns those scores into probabilities. The next question is how well those probabilities fit text held out from training.

Mastery check

What strong answers show

• Foundational: You can explain why an embedding lookup returns one stored row for every occurrence of a token ID.
• Intermediate: You can build a co-occurrence or prediction example and show why the two meanings of charge require context mixing.
• Advanced: You can choose a representation for a measured failure case and test whether cosine geometry supports the chosen metric.

Follow-up questions

Why can't a static embedding table represent billing charge and device charge differently in these two sentences?

Answer

The tokenizer emits the same token ID for both occurrences, and a static lookup retrieves one stored row per ID. Only a later context-dependent computation can use words such as invoice or scanner to separate the senses.

What does a FastText-style character n-gram representation fix for tokenized, and what does it not fix for charge?

Answer

Character n-grams let an unseen spelling such as tokenized reuse pieces learned from related forms such as tokenize. They don't decide whether an already-known spelling such as charge means a bill or a powered device in a particular sentence.

Why should a retrieval team test cosine similarity on held-out queries before trusting contextual vectors?

Answer

Contextual vectors can share dominant directions, so unrelated token states may receive high raw cosine scores. Held-out retrieval or classification examples reveal whether the geometry supports the product decision.

When embeddings mislead

• Treating token IDs as meaning: An ID only selects a row. Training and context create useful geometry.
• Calling subword handling disambiguation: Character pieces help unfamiliar spellings, not multiple senses of one familiar word.
• Trusting cosine without an evaluation set: A high score can come from shared directions rather than task-relevant similarity.