Static to Contextual Embeddings

The previous lesson turned text into token IDs. Those IDs are model-local row indices, not meanings. The same ID can appear in a billing dispute and in a device instruction:

"dispute the charge" -> [91, 12, 407]
"charge the scanner"  -> [407, 12, 982]

These are invented IDs for a word-level tokenizer. Both lines contain ID 407, yet dispute and scanner point to different uses. If both lookups start from one row, where can that difference enter? An embedding stores a vector of numbers for each ID. A contextual representation can then rewrite that vector using words around this particular occurrence.

First inspect the static table. Then watch the same token leave with two different states.

The first stop is the lookup. Its limitation gives context mixing a job.

A token ID selects one stored row

Let vocabulary size be V and embedding dimension be d. The model stores an embedding matrix E with shape V x d.

The table answers one narrow question: given an ID, which vector enters the model?

• Row E[token_id] is the starting vector for that token.
• Training adjusts rows to support its objective. Similar usage can lead to related vectors, but training doesn't guarantee a useful cosine neighborhood.
• Before any context mixing, every occurrence of the same ID receives that same row.

A one-hot row vector of length V with a 1 at token_id selects the same row. Multiplying it by E writes the lookup as matrix algebra.

If charge is ID 0 in a tiny table, both billing and device sentences start at E[0]. A production tokenizer might use 407 instead. The rule doesn't change: the integer is an index, not a sense label.

The lab uses a three-row table so every index is visible. Before running it, predict whether the two printed rows can differ.

vocab = {"charge": 0, "invoice": 1, "scanner": 2}
embedding_table = [
    [0.80, 0.20],  # charge
    [0.20, 1.00],  # invoice
    [1.00, 0.10],  # scanner
]

billing_charge = embedding_table[vocab["charge"]]
device_charge = embedding_table[vocab["charge"]]

print("billing start:", billing_charge)
print("device start:", device_charge)
print("same starting row:", billing_charge == device_charge)

billing start: [0.8, 0.2]
device start: [0.8, 0.2]
same starting row: True

The rows are equal, so no lookup-only operation can resolve the two senses. Here, static means independent of the surrounding sentence, not permanently frozen: training can update the row. Position information may also change a model's input state, but position alone doesn't identify a word's meaning.

Nearby words leave a fingerprint

If encoder and decoder keep seeing layer and mask, their neighborhood fingerprints start to resemble each other. The same logic puts cache near prefix when both occur with reuse language. This is the distributional hunch: usage supplies a weak signal about role.

The simplest measurement is a co-occurrence count. For each token, count words inside a fixed context window.

from collections import Counter, defaultdict

messages = [
    "encoder layer uses attention mask".split(),
    "decoder layer uses attention mask".split(),
    "encoder block mixes token context".split(),
    "cache hit reuses prompt prefix".split(),
    "prefix cache skips prompt compute".split(),
]

window = 2
neighbors = defaultdict(Counter)

for message in messages:
    for center_index, center in enumerate(message):
        left = max(0, center_index - window)
        right = min(len(message), center_index + window + 1)
        for context_index in range(left, right):
            if context_index != center_index:
                neighbors[center][message[context_index]] += 1

print("encoder context:", neighbors["encoder"].most_common(4))
print("decoder context:", neighbors["decoder"].most_common(4))
print("cache context:", neighbors["cache"].most_common(4))

encoder context: [('layer', 1), ('uses', 1), ('block', 1), ('mixes', 1)]
decoder context: [('layer', 1), ('uses', 1)]
cache context: [('hit', 1), ('reuses', 1), ('prefix', 1), ('skips', 1)]

The sample exposes the fingerprints: encoder and decoder share layer and uses, while cache sees prefix and reuse language. The rows are still counts, not compact coordinates.

A count row is high-dimensional and noisy. Latent semantic analysis compressed a term-document matrix with singular value decomposition (SVD).^{[1]Reference 1Indexing by Latent Semantic Analysishttps://doi.org/10.1002/(SICI)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9}

The same move works on word-context counts: keep directions that explain the main neighborhoods, drop the rest.

Raw counts can mistake a common word for an informative neighbor. Positive pointwise mutual information (PPMI) keeps pairs that occur more often than chance and floors negative values at zero.

For example, suppose a table contains 53 word-context pairs in total. The encoder row totals 14 and the layer column totals 15. Independent occurrence would predict 14 × 15 / 53 ≈ 3.96 pairs. Observing 8 gives a ratio of 8 / 3.96 ≈ 2.02, whose natural logarithm is about 0.70. That positive value records a stronger association than the marginals alone predict.

If #(w, c) is the count for word w with context c,

where N is the total of all counts, #(w) is the row sum, and #(c) is the column sum.

After compression, cosine similarity asks whether two vectors point the same way, ignoring length:

For nonzero vectors, parallel directions score 1, right angles score 0, and opposite directions score -1. A zero vector has no direction, so its cosine is undefined. For [1, 0] and [1, 1], the dot product is 1 and the lengths are 1 and √2, giving cosine 1 / √2 ≈ 0.707.

The next lab uses NumPy and a deliberately separate 4×4 fixture, not counts from the five messages above. Architecture words (encoder, decoder) share layer and mask; serving words (cache, prefix) share hit and reuse. It converts counts to PPMI, then keeps two SVD components.

SVD writes the PPMI matrix as $U\Sigma V^T$. Columns of $U$ locate words along the learned directions, and the diagonal of $\Sigma$ measures each direction's strength. We use word vectors $U_2\sqrt{\Sigma_2}$, splitting the strength between word and context factors. These coordinates aren't uniquely signed: flipping a whole column preserves the geometry.

Predict the result before running it: which pair should have cosine 1, and which pair should be separated?

import numpy as np

words = ["encoder", "decoder", "cache", "prefix"]
counts = np.array([
    [8, 6, 0, 0],  # encoder
    [7, 6, 0, 0],  # decoder
    [0, 0, 8, 5],  # cache
    [0, 0, 7, 6],  # prefix
], dtype=float)

def cosine(a, b):
    denominator = np.linalg.norm(a) * np.linalg.norm(b)
    if denominator == 0:
        raise ValueError("Cosine requires two nonzero vectors")
    return float(a @ b / denominator)

def ppmi_matrix(table):
    if table.ndim != 2 or np.any(table < 0) or not np.all(np.isfinite(table)):
        raise ValueError("Counts must be a finite nonnegative matrix")
    result = np.zeros_like(table, dtype=float)
    if table.sum() == 0:
        return result
    expected = table.sum(axis=1, keepdims=True) @ table.sum(axis=0, keepdims=True)
    expected /= table.sum()
    observed = table > 0
    result[observed] = np.maximum(np.log(table[observed] / expected[observed]), 0)
    return result

ppmi = ppmi_matrix(counts)
left, singular, right_transpose = np.linalg.svd(ppmi, full_matrices=False)
vectors = left[:, :2] * np.sqrt(singular[:2])
assert vectors.shape == (len(words), 2)
assert np.all(ppmi_matrix(np.zeros((2, 2))) == 0)

print("encoder-layer PPMI:", round(float(ppmi[0, 0]), 3))
print("singular values:", round(float(singular[0]), 3), round(float(singular[1]), 3))
print("encoder vs decoder:", round(cosine(vectors[0], vectors[1]), 3))
print("encoder vs cache:", round(cosine(vectors[0], vectors[2]), 3))

encoder-layer PPMI: 0.703
singular values: 1.418 1.348
encoder vs decoder: 1.0
encoder vs cache: 0.0

The architecture rows become parallel. The serving rows land on a second axis at right angles to the first. Similar neighborhoods became similar directions, which is the distributional hunch made geometric.

The constructed blocks make the split obvious. Real corpora add messy wording, rare contexts, and words that play several roles. A rare pair can also get high PPMI from very little evidence, so a high association score isn't a confidence estimate. Counting is one route to this geometry, not the only route.

Predict the neighbors instead

Counting is explicit. Word2Vec learns the same kind of static table by trying to predict neighbors:

• CBOW (continuous bag-of-words) predicts a center token from its surrounding tokens.
• Skip-gram predicts surrounding tokens from a center token.

There is no label saying “similar.” Each correct prediction nudges vectors so tokens that support similar predictions can share a neighborhood. Mikolov and colleagues introduced the two architectures in 2013.^{[2]Reference 2Efficient Estimation of Word Representations in Vector Space.https://arxiv.org/abs/1301.3781}

Skip-gram turns each window into training pairs. With window size 2, center hit produces one positive pair for each visible neighbor. The center stays first in each pair.

Count those pairs before looking at the output: the window reaches request, cache, and layer, but not today.

tokens = "request hit cache layer today".split()
window = 2
pairs = []

for center_index, center in enumerate(tokens):
    left = max(0, center_index - window)
    right = min(len(tokens), center_index + window + 1)
    for context_index in range(left, right):
        if context_index != center_index:
            pairs.append((center, tokens[context_index]))

hit_pairs = [pair for pair in pairs if pair[0] == "hit"]
print(hit_pairs)

[('hit', 'request'), ('hit', 'cache'), ('hit', 'layer')]

The output confirms the boundary: today is three positions away, so it never becomes a positive pair for hit. Each listed pair supplies one local prediction target.

Scoring every vocabulary item for every pair is expensive. A Skip-gram trainer can instead treat the observed neighbor as a positive example and draw a few vocabulary items from a noise distribution as negative samples. A sampled item isn't necessarily absent from the corpus or semantically unrelated; its negative label belongs to this training draw. The original negative-sampling paper uses unigram frequency raised to the 3/4 power, then normalized.^{[3]Reference 3Distributed Representations of Words and Phrases and their Compositionalityhttps://arxiv.org/abs/1310.4546} The next lab supplies its samples directly.

Skip-gram maintains separate input and output tables: $u$ is the center's input vector, $v_+$ is the observed neighbor's output vector, and $v_n$ is a sampled item's output vector. The loss rewards a high center-neighbor score and low center-negative scores. Here $\sigma$ is the logistic sigmoid, and $\mathcal{N}$ is the multiset of sampled negatives, retaining repeated draws.^{[4]Reference 4Neural Word Embedding as Implicit Matrix Factorization.https://proceedings.neurips.cc/paper_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html}

A high positive dot product lowers the first term. A negative dot product lowers each negative-sample term. This constrains input-output scores; it doesn't directly require every observed pair to have similar input vectors.

With center [1, 0] and observed output [1.2, 0.1], the positive score is 1.2. Holding output vectors fixed, one gradient step changes the center. Predict whether that step raises the positive score and lowers the loss. The code uses $\operatorname{softplus}(x)=\log(1+e^x)$, evaluated without overflowing the exponential. The positive term becomes $\operatorname{softplus}(-u\cdot v_+)$.

The loss derivative with respect to a positive score is sigmoid(score) - 1; for a negative score it's sigmoid(score). Multiplying each derivative by its output vector and adding gives the center gradient. Subtracting 0.1 × gradient takes a small downhill step.

import math

center = [1.0, 0.0]
observed_neighbor = [1.2, 0.1]
negative_samples = [[-0.9, -0.2], [-1.1, 0.0]]

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def sigmoid(x):
    if x >= 0:
        return 1.0 / (1.0 + math.exp(-x))
    exp_x = math.exp(x)
    return exp_x / (1.0 + exp_x)

def softplus(x):
    return max(x, 0.0) + math.log1p(math.exp(-abs(x)))

def negative_sampling_loss(center_vector, positive_vector, negatives):
    positive_loss = softplus(-dot(center_vector, positive_vector))
    negative_loss = sum(
        softplus(dot(center_vector, sample))
        for sample in negatives
    )
    return positive_loss + negative_loss

before = negative_sampling_loss(center, observed_neighbor, negative_samples)
positive_error = sigmoid(dot(center, observed_neighbor)) - 1.0
gradient = [
    positive_error * observed_neighbor[j]
    + sum(sigmoid(dot(center, sample)) * sample[j] for sample in negative_samples)
    for j in range(len(center))
]
updated = [value - 0.1 * grad for value, grad in zip(center, gradient)]
after = negative_sampling_loss(updated, observed_neighbor, negative_samples)
assert after < before
assert softplus(1000.0) == 1000.0
assert softplus(-1000.0) == 0.0  # The tiny positive value rounds to zero.

print("center after one step:", [round(value, 3) for value in updated])
print("positive score:", round(dot(center, observed_neighbor), 3), "->",
      round(dot(updated, observed_neighbor), 3))
print("loss:", round(before, 3), "->", round(after, 3))

center after one step: [1.081, 0.008]
positive score: 1.2 -> 1.298
loss: 0.892 -> 0.827

Only the center moved in this demonstration; a full trainer updates the participating output rows too. Repeating such updates can make input words with similar neighbor predictions acquire similar vectors.

Levy and Goldberg connected this objective to a shifted PMI matrix. When negatives are drawn in proportion to their context frequency, the unconstrained optimum for an observed input-output pair's dot product is $\operatorname{PMI}(w,c)-\log k$, where $k$ is the number of negative draws per positive pair. An unobserved pair with positive negative-sampling probability has its ideal score at negative infinity. Finite dimension, regularization, and the original 3/4-power noise distribution change the fit, so this isn't an assertion that SVD and Word2Vec return identical vectors.^{[4]Reference 4Neural Word Embedding as Implicit Matrix Factorization.https://proceedings.neurips.cc/paper_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html}

GloVe aggregates local context-window observations into one corpus-wide count table. “Global” describes that aggregation; it doesn't mean pairing words at arbitrary distances.

Fit the global count table

GloVe starts from those global co-occurrence counts. Its original experiments also weight a pair at distance $r$ within the window by $1/r$, so an entry can be fractional. It fits a weighted least-squares objective: a word vector and a context vector, plus two biases, should reconstruct the logarithm of each observed count.^{[5]Reference 5GloVe: Global Vectors for Word Representation.https://aclanthology.org/D14-1162.pdf}

The dot product is the model's compatibility score for that word-context pair.

The logarithm compresses counts that span many orders of magnitude. A weighting function then limits how much very common pairs dominate:

The paper's experiments use $x_{\max} = 100$ and $\alpha = 3/4$. Rare pairs get small weights. Very frequent pairs stop gaining extra weight past the cutoff.

Only observed, nonzero counts enter this calculation. Evaluating log(0) and multiplying it by zero afterward is numerically invalid; skip those entries before taking logs.

The lab fixes the model score at log(30). Predict which count has zero residual, then see how weighting changes with frequency.

import math

def squared_glove_residual(observed_count, model_score):
    target = math.log(observed_count)
    return (model_score - target) ** 2

def glove_weight(count, xmax=100, alpha=0.75):
    if count < xmax:
        return (count / xmax) ** alpha
    return 1.0

model_score = math.log(30)
matching_error = squared_glove_residual(30, model_score)
different_error = squared_glove_residual(3, model_score)

print("log targets:", [round(math.log(count), 3) for count in (3, 30, 300)])
print("matching count error:", round(matching_error, 3))
print("different count error:", round(different_error, 3))
print("weights for 3, 30, 100:", [round(glove_weight(count), 3) for count in (3, 30, 100)])

log targets: [1.099, 3.401, 5.704]
matching count error: 0.0
different count error: 5.302
weights for 3, 30, 100: [0.072, 0.405, 1.0]

The score matches log(30), so count 30 has zero residual. Count 3 misses that target and pays a residual of 5.302; the weighting function also gives rarer pairs less influence.

Word2Vec and GloVe still give each known spelling one static row. That makes lookup cheap, but leaves two gaps: a new spelling has no row, and an ambiguous word has only one.

Build unknown spellings from pieces

Technical text constantly produces variants: tokenize, tokenized, tokenizer, plus typos mixed into logs. FastText represents a known word with character n-grams as well as its whole-word identity, so related spellings share parameters.^{[6]Reference 6Enriching Word Vectors with Subword Information.https://arxiv.org/abs/1607.04606}

The paper uses n-gram lengths 3 through 6 with boundary markers. The lab uses length 3 so every window is visible.

For an out-of-vocabulary (OOV) spelling, there's no learned whole-word row. FastText can still combine its n-gram bucket rows. This produces a representation without guaranteeing useful meaning: some buckets may be poorly trained, and different strings can collide in one bucket.

The paper describes a sum in its scoring function. The official getWordVector API sums component rows and then averages them by their count.^{[7]Reference 7fastText implementation: getWordVector component averaginghttps://github.com/facebookresearch/fastText/blob/main/src/fasttext.cc} A positive rescaling preserves a nonzero vector's cosine direction, but changes its norm and dot products.

def character_ngrams(word, n=3):
    marked = f"<{word}>"
    return {marked[i:i + n] for i in range(len(marked) - n + 1)}

known = character_ngrams("tokenize")
new_form = character_ngrams("tokenized")
shared = sorted(known & new_form)

print("shared pieces:", shared)
print("new spelling can reuse pieces:", len(shared) > 0)
print("tokenize-only pieces:", sorted(known - new_form))
print("tokenized-only pieces:", sorted(new_form - known))

shared pieces: ['<to', 'eni', 'ize', 'ken', 'niz', 'oke', 'tok']
new spelling can reuse pieces: True
tokenize-only pieces: ['ze>']
tokenized-only pieces: ['ed>', 'zed']

The output isolates the reusable pieces. A trained FastText model combines vectors for all the spelling's n-grams, not just the overlap shown here. It maps n-grams to a fixed set of hash buckets, so distinct pieces can share a row too. For unseen tokenized, it skips the whole-word row. The code only compares unique strings; it isn't a full FastText tokenizer or vector lookup. The implementation retains repeated n-gram occurrences, so using this set-based lab as an encoder would also change their weights.^{[6]Reference 6Enriching Word Vectors with Subword Information.https://arxiv.org/abs/1607.04606[7]Reference 7fastText implementation: getWordVector component averaginghttps://github.com/facebookresearch/fastText/blob/main/src/fasttext.cc}

OOV composition needs an artifact containing those component rows, such as the full .bin model with subwords enabled. A .vec export holds vectors for vocabulary words and doesn't provide the subword model needed to compose new spellings.^{[8]Reference 8Word representations: full model versus exported word vectorshttps://fasttext.cc/docs/en/unsupervised-tutorial.html}

Subword composition helps with unfamiliar surface forms. It still can't decide which meaning an already-known ambiguous token carries.

One static row can't choose a sense

Read these two messages:

"Dispute the charge on invoice 8142."
"Charge the scanner before the lab demo."

Assume the tokenizer lowercases both occurrences to the same charge ID. A static lookup returns the same vector, even though the first case belongs near billing language and the second near device language.

Before running the lab, predict whether either similarity score can notice that difference.

import math

static = {
    "charge": [0.8, 0.2],
    "invoice": [0.2, 1.0],
    "scanner": [1.0, 0.1],
}

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def cosine(a, b):
    return dot(a, b) / math.sqrt(dot(a, a) * dot(b, b))

billing_charge = static["charge"]
device_charge = static["charge"]

print("same charge vector:", billing_charge == device_charge)
print("billing similarity to invoice:", round(cosine(billing_charge, static["invoice"]), 3))
print("device similarity to invoice:", round(cosine(device_charge, static["invoice"]), 3))

same charge vector: True
billing similarity to invoice: 0.428
device similarity to invoice: 0.428

Both comparisons match because the two charge vectors are identical. A static row can encode several useful associations, but it can't select a different vector for this occurrence's sense. It isn't necessarily the arithmetic midpoint of sense-specific vectors. Sentence evidence has to enter after the lookup.

Mix in a neighbor from this sentence

For a fixed set of model weights, the lookup stays the same across sentences, so context enters in layers above it. A Transformer can learn a weighted mix of neighboring states.

To isolate that move, take the shared charge row and average it with one clue from its sentence: billing with invoice, device with scanner.

The two starts are identical. Predict where the final states should move before running the lab.

import math

table = {
    "charge": [0.8, 0.2],
    "invoice": [0.2, 1.0],
    "scanner": [1.0, 0.1],
}

def mix(token_vector, clue_vector):
    return [0.5 * a + 0.5 * b for a, b in zip(token_vector, clue_vector)]

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def cosine(a, b):
    return dot(a, b) / math.sqrt(dot(a, a) * dot(b, b))

billing_state = mix(table["charge"], table["invoice"])
device_state = mix(table["charge"], table["scanner"])

print("billing charge state:", [round(x, 2) for x in billing_state])
print("device charge state:", [round(x, 2) for x in device_state])
print("same final state:", billing_state == device_state)
print("billing vs invoice:", round(cosine(billing_state, table["invoice"]), 3))
print("device vs invoice:", round(cosine(device_state, table["invoice"]), 3))

billing charge state: [0.5, 0.6]
device charge state: [0.9, 0.15]
same final state: False
billing vs invoice: 0.879
device vs invoice: 0.355

The equal weights and all starting coordinates are hand-assigned. Billing charge moves toward invoice (cosine 0.879), while device charge stays farther away (0.355). This isolates context dependence without training attention. A real attention head mixes learned value projections from allowed positions; residual paths, output projections, and other layers further transform the state. Comparing its output directly with input-table rows doesn't automatically measure sense quality.

The 2018 ELMo (Embeddings from Language Models) model combines forward and backward recurrent language models. Their input representation and output softmax parameters are shared; their long short-term memory (LSTM) parameters are separate. A downstream task learns a weighted mix of layer states, ordinarily with the pretrained language-model weights frozen.^{[9]Reference 9Deep contextualized word representations.https://arxiv.org/abs/1802.05365}

At position k, $h_{k,0}$ comes from a context-free character convolutional neural network (CNN). The original model projects this representation to 512 dimensions; the implementation duplicates it to 1,024 dimensions for the layer mix. Each contextual state $h_{k,1}$ and $h_{k,2}$ concatenates 512-dimensional forward and backward LSTM outputs, so all three shapes match.^{[9]Reference 9Deep contextualized word representations.https://arxiv.org/abs/1802.05365[10]Reference 10AllenNLP ELMo implementation: context-free layer duplicationhttps://github.com/allenai/allennlp/blob/main/allennlp/modules/elmo.py} Their task-specific combination is

The task-specific weights $s_j$ are softmax-normalized, making them nonnegative and summing to 1. They are shared across token positions for that task, rather than being token-specific attention weights. $\gamma$ scales the whole mix. Lower and upper layers can therefore contribute different kinds of information.

BERT (Bidirectional Encoder Representations from Transformers) changed how deep bidirectional context is trained. Its masked language-model objective corrupts selected input tokens and predicts their originals using both sides. Encoder self-attention can therefore read evidence before and after a position.^{[11]Reference 11BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}

The binary visibility mask says who may attend. Learned attention weights say how much each allowed token contributes. Those are separate objects.

Not every model may look both ways

A BERT-style encoder token can use tokens on both sides during encoding. A GPT-style causal language model predicts the next token from the prefix, so its state at a position can't use later tokens while preserving that objective.

The original GPT work used a Transformer decoder for generative pretraining.^{[12]Reference 12Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}

At the first token in charge the scanner, the right-hand clue scanner is visible to an encoder and hidden from a causal decoder state at charge. Predict which list each mask should produce.

tokens = ["charge", "the", "scanner"]
encoder_visibility = [
    [1, 1, 1],
    [1, 1, 1],
    [1, 1, 1],
]
causal_visibility = [
    [1, 0, 0],
    [1, 1, 0],
    [1, 1, 1],
]

def visible_tokens(mask, position):
    return [token for token, allowed in zip(tokens, mask[position]) if allowed]

print("encoder state for charge sees:", visible_tokens(encoder_visibility, 0))
print("causal state for charge sees:", visible_tokens(causal_visibility, 0))
print("causal state for scanner sees:", visible_tokens(causal_visibility, 2))

alternative = ["charge", "the", "invoice"]
original_prefix = visible_tokens(causal_visibility, 0)
alternative_prefix = [
    token for token, allowed in zip(alternative, causal_visibility[0]) if allowed
]
print("same visible prefix after replacing scanner:", original_prefix == alternative_prefix)

encoder state for charge sees: ['charge', 'the', 'scanner']
causal state for charge sees: ['charge']
causal state for scanner sees: ['charge', 'the', 'scanner']
same visible prefix after replacing scanner: True

Both designs produce contextual states. Their visibility rules match different training objectives: one can use the whole input, while the other must preserve prefix-only prediction.

The causal state at scanner can use the whole three-token prefix, so it can incorporate the device clue. The earlier state at charge still can't use it. With the same prefix token IDs, positions and fixed weights, replacing a later token leaves that earlier state unchanged in deterministic evaluation. Processing the full sentence in parallel doesn't remove the causal mask. If you need a representation of the whole message, extracting the state at its first ambiguous word would discard the later evidence.

Even when two states are allowed to differ, a similarity score can still lie to you.

A shared direction can fake similarity

Embedding applications often use cosine similarity: a value near 1 means two vectors point in similar directions. That score can mislead when the space shares a large, uninformative direction.

A 2019 study of BERT, ELMo, and GPT-2 found that many layers were anisotropic, with vectors occupying a narrow cone and raw cosine high even between unrelated sampled words.^{[13]Reference 13How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.https://arxiv.org/abs/1909.00512} Anisotropy means the distribution isn't directionally uniform; a shared mean is one possible cause. It isn't a universal diagnosis of every current embedding model.

Two states can disagree in a task-relevant component while still pointing almost the same way overall.

Predict what raw cosine will say before running the lab. The two states share a large horizontal component, so raw cosine treats them as almost identical. Subtracting their shared mean exposes vertical components that point in opposite directions.

import math

billing = [10.0, 1.0]
device = [10.0, -1.0]

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def cosine(a, b):
    return dot(a, b) / math.sqrt(dot(a, a) * dot(b, b))

raw_similarity = cosine(billing, device)
mean_direction = [(a + b) / 2 for a, b in zip(billing, device)]
centered_billing = [a - m for a, m in zip(billing, mean_direction)]
centered_device = [a - m for a, m in zip(device, mean_direction)]
centered_similarity = cosine(centered_billing, centered_device)

print("raw cosine:", round(raw_similarity, 3))
print("centered cosine:", round(centered_similarity, 3))

raw cosine: 0.98
centered cosine: -1.0

The raw cosine is 99 / 101 ≈ 0.98. Subtracting [10, 0] leaves [0, 1] and [0, -1], whose cosine is -1. But centering any two distinct vectors around their own midpoint makes them opposites. That result alone doesn't prove that a real embedding population is anisotropic or that centering improves retrieval.

For a dataset-level diagnostic, estimate a mean from a representative training sample and apply that same mean to held-out vectors. Then measure retrieval or classification quality. Don't fit a new mean separately to each query-document pair, and don't treat the constructed vertical axis as a discovered meaning dimension.

Match the representation to the failure

Choose a representation from the failure mode and the measurement that could prove an improvement. No representation wins by name alone.

Contextual token states aren't automatically good message-level retrieval vectors. Pooling and task training change what a vector means, so evaluate those choices on the retrieval task. Sentence-BERT's original experiments showed why averaging raw BERT states or taking its [CLS] output wasn't enough for useful cosine similarity on their semantic-similarity tasks.^{[14]Reference 14Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084} That historical result doesn't establish a universal ranking of present-day models.

Keep query and document coordinates compatible

A search index may accept any vector of the right length. That doesn't make its numbers comparable with a query from a different model.

Imagine two perfect copies of our small space, with the second rotated 90 degrees. Either space preserves its own cosine relationships. Mixing a query from the first with documents from the second can reverse the ranking. Predict which document wins in each case:

import math

query_a = [1.0, 0.0]  # Authored billing query
documents_a = {"invoice": [1.0, 0.0], "scanner": [0.0, -1.0]}

def rotate(vector):
    x, y = vector
    return [-y, x]

def cosine(a, b):
    denominator = math.sqrt(sum(x*x for x in a) * sum(x*x for x in b))
    if denominator == 0:
        raise ValueError("Cosine requires nonzero vectors")
    return sum(x*y for x, y in zip(a, b)) / denominator

documents_b = {name: rotate(vector) for name, vector in documents_a.items()}
query_b = rotate(query_a)
def top_document(query, documents):
    return max(documents, key=lambda name: cosine(query, documents[name]))

print("A query, A index:", top_document(query_a, documents_a))
print("A query, B index:", top_document(query_a, documents_b))
print("B query, B index:", top_document(query_b, documents_b))
assert cosine(query_a, documents_a["invoice"]) == cosine(query_b, documents_b["invoice"])
assert top_document(query_a, documents_b) == "scanner"

A query, A index: invoice
A query, B index: scanner
B query, B index: invoice

This rotation is a constructed counterexample, not an actual model migration. It proves that matching dimension and normalizing length don't establish coordinate compatibility. Two independently trained models needn't differ by a simple rotation.

Use a verified query/document encoder pair and its prescribed preprocessing, pooling, dimension, and similarity metric. The pair can use jointly trained encoders with separate weights, as in Dense Passage Retrieval.^{[15]Reference 15Dense Passage Retrieval for Open-Domain Question Answering.https://arxiv.org/abs/2004.04906} Compatible doesn't always mean identical encoding calls either. Current Sentence Transformers search examples use encode_query and encode_document for those roles.^{[16]Reference 16Semantic Search: query and document encodinghttps://www.sbert.net/examples/sentence_transformer/applications/semantic-search/README.html} Follow the selected model's instructions for prompts or prefixes, pin its revision and configuration, and rebuild the document index when changing the space unless a validated migration preserves compatibility.

In a causal language model, the current position's contextual state feeds the output head, which scores candidate next tokens and turns those scores into probabilities. The next question is how well those probabilities fit text held out from training.

Evaluation rubric

• Explain why two occurrences of one token ID begin with the same embedding row but can leave a contextual layer with different states.
• Compute a skip-gram score and distinguish the training objective from a cosine-similarity lookup.
• Diagnose misleading cosine neighbors caused by a shared direction, and evaluate a proposed correction on the intended task.

Follow-up questions