LearnCore LLM FoundationsStatic to Contextual Embeddings

📝MediumNLP Fundamentals

Static to Contextual Embeddings

Turn token IDs into vectors, learn what nearby usage captures, and see why a word such as charge needs sentence-dependent representations.

13 min read

Learning path

Step 47 of 155 in the full curriculum

BPE, WordPiece, and SentencePiece Perplexity & Model Evaluation

A tokenizer can turn a support request into IDs:

text

"dispute the charge" -> [91, 12, 407]
"charge the scanner"  -> [407, 12, 982]

IDs solve naming. They don't tell a model that refund resembles return, or that charge means a billing event in one message and supplying power in another.

An embedding gives each token a of numbers. A contextual representation goes one step further: it adjusts that vector using the sentence around this particular occurrence.

By the end of this lesson, you'll build both ideas from scratch and watch the same leave with two different meanings.

Diagram showing Support message, Tokenizer token IDs, Embedding table one starting row per ID, and Context-mixing layers tokens exchange evidence. — Support message, Tokenizer token IDs, Embedding table one starting row per ID, and Context-mixing layers tokens exchange evidence.

Look Up Coordinates for Token IDs

Suppose a tokenizer has a vocabulary of size V, and you choose embedding dimension d. The model stores an embedding E with shape V x d.

Row E[token_id] is the starting vector for that token.
Training changes rows so tokens used in similar situations become useful to the prediction task.
Before context mixing, every occurrence of the same token ID receives the same row.

That final point matters. If charge is ID 0, both billing and device sentences initially retrieve row E[0].

01-look-up-token-rows.py

import numpy as np

vocab = {"charge": 0, "refund": 1, "scanner": 2}
embedding_table = np.array([
    [0.80, 0.20],  # charge
    [0.15, 0.95],  # refund
    [0.92, 0.10],  # scanner
])

billing_charge = embedding_table[vocab["charge"]]
device_charge = embedding_table[vocab["charge"]]

print("billing start:", billing_charge.tolist())
print("device start:", device_charge.tolist())
print("same starting row:", np.array_equal(billing_charge, device_charge))

Output

billing start: [0.8, 0.2]
device start: [0.8, 0.2]
same starting row: True

A lookup table alone can't distinguish senses. First, though, it has to learn a useful map of token usage.

Learn Meaning from Neighboring Words

Words that repeatedly appear near similar words tend to play similar roles. In return workflows, refund and replacement may both occur near approved, request, or label; carrier and tracking may cluster around shipment status language.

The simplest evidence is a co-occurrence count: for every token, count words within a fixed context window.

Static versus contextual embeddings for the word charge, showing one Word2Vec lookup vector versus separate billing-charge and scanner-charge vectors shaped by surrounding words. — A stored charge row is identical until sentence evidence turns each occurrence into a different final state.

02-count-context-neighbors.py

from collections import Counter, defaultdict

messages = [
    "refund approved send return label".split(),
    "replacement approved send return label".split(),
    "refund request needs label".split(),
    "tracking update from carrier hub".split(),
    "shipment delayed at carrier hub".split(),
]

window = 2
neighbors = defaultdict(Counter)

for message in messages:
    for center_index, center in enumerate(message):
        left = max(0, center_index - window)
        right = min(len(message), center_index + window + 1)
        for context_index in range(left, right):
            if context_index != center_index:
                neighbors[center][message[context_index]] += 1

print("refund context:", neighbors["refund"].most_common(4))
print("replacement context:", neighbors["replacement"].most_common(4))
print("carrier context:", neighbors["carrier"].most_common(4))

Output

refund context: [('approved', 1), ('send', 1), ('request', 1), ('needs', 1)]
replacement context: [('approved', 1), ('send', 1)]
carrier context: [('hub', 2), ('update', 1), ('from', 1), ('delayed', 1)]

A count row can be high-dimensional and noisy. One historical route, latent semantic analysis, uses singular value decomposition (SVD) to compress a high-dimensional term-document matrix.^[1] For this lesson, use the same compression move on word-context counts. The compressed rows act as static vectors: one vector per word, independent of sentence.

Count-based embedding pipeline showing a sparse word-context matrix, SVD factorization, rank-k truncation, and dense word vectors. — SVD compresses a sparse context table so return tokens and shipment tokens occupy different nearby regions.

The next exercise follows that word-context route. It computes positive pointwise mutual information (PPMI), which emphasizes pairs occurring more often than chance, then compresses the matrix with SVD.

03-compress-ppmi-with-svd.py

import numpy as np

words = ["refund", "replacement", "carrier", "tracking"]
contexts = ["label", "approved", "hub", "delayed"]
counts = np.array([
    [8, 6, 0, 0],  # refund
    [7, 6, 0, 0],  # replacement
    [0, 0, 8, 5],  # carrier
    [0, 0, 7, 6],  # tracking
], dtype=float)

expected = counts.sum(axis=1, keepdims=True) @ counts.sum(axis=0, keepdims=True) / counts.sum()
ppmi = np.maximum(np.log((counts + 1e-9) / (expected + 1e-9)), 0.0)
u, singular_values, _ = np.linalg.svd(ppmi, full_matrices=False)
vectors = u[:, :2] * np.sqrt(singular_values[:2])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print("refund vs replacement:", round(cosine(vectors[0], vectors[1]), 3))
print("refund vs carrier:", round(cosine(vectors[0], vectors[2]), 3))

Output

refund vs replacement: 1.0
refund vs carrier: 0.0

This tiny dataset is designed to make the pattern visible. Real corpora have far more contexts, imperfect wording, and competing meanings.

Toy static embedding map showing return-related and shipment-related words clustering because their context neighborhoods overlap. — Nearby coordinates reflect overlapping usage in this toy corpus, not a universal semantic map.

Predict Neighbors with Word2Vec

Counting first and compressing later isn't the only way to learn a static embedding table. Word2Vec trains vectors through prediction:

CBOW predicts a center token from its surrounding tokens.
Skip-gram predicts surrounding tokens from a center token.

Both objectives reward vectors that help predict observed neighborhoods. Mikolov and colleagues introduced these two architectures in 2013.^[2]

Word2Vec CBOW and skip-gram objectives showing context tokens predicting a center word versus a center word predicting surrounding context tokens. — Both Word2Vec objectives learn from observed windows; they reverse which side supplies the prediction target.

For Skip-gram, a training set can be built directly from windows. With window size 2, the center arrived produces one positive pair for each visible neighbor.

04-create-skipgram-pairs.py

tokens = "package arrived at carrier hub today".split()
window = 2
pairs = []

for center_index, center in enumerate(tokens):
    left = max(0, center_index - window)
    right = min(len(tokens), center_index + window + 1)
    for context_index in range(left, right):
        if context_index != center_index:
            pairs.append((center, tokens[context_index]))

arrived_pairs = [pair for pair in pairs if pair[0] == "arrived"]
print(arrived_pairs)

Output

[('arrived', 'package'), ('arrived', 'at'), ('arrived', 'carrier')]

Predicting every vocabulary item for every pair is costly. Negative sampling trains the observed pair as positive and a small set of sampled unobserved pairs as negative. The following objective is a small, inspectable version of that idea.

05-compare-negative-sampling-loss.py

import numpy as np

center = np.array([1.0, 0.0])
observed_neighbor = np.array([1.2, 0.1])
wrong_neighbor = np.array([-1.0, 0.1])
negative_samples = [np.array([-0.9, -0.2]), np.array([-1.1, 0.0])]

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

def negative_sampling_loss(center_vector, positive_vector, negatives):
    positive_loss = -np.log(sigmoid(center_vector @ positive_vector))
    negative_loss = sum(-np.log(sigmoid(-(center_vector @ n))) for n in negatives)
    return float(positive_loss + negative_loss)

observed_loss = negative_sampling_loss(center, observed_neighbor, negative_samples)
wrong_loss = negative_sampling_loss(center, wrong_neighbor, negative_samples)

print("observed pair loss:", round(observed_loss, 3))
print("wrong pair loss:", round(wrong_loss, 3))
print("training prefers observed pair:", observed_loss < wrong_loss)

Output

observed pair loss: 0.892
wrong pair loss: 1.942
training prefers observed pair: True

No analogy trick is required to understand the useful result: after enough examples, tokens that support similar neighbor predictions can end up near one another.

Fit Global Counts with GloVe

GloVe starts from global co-occurrence counts. It fits a weighted least-squares objective: a word-vector and context-vector dot product, plus biases, should approximate the logarithm of each observed count. The logarithm compresses the count range, while the weighting function limits how much very common pairs dominate training.^[3]

06-fit-glove-log-count.py

import numpy as np

def squared_glove_residual(observed_count, model_score):
    target = np.log(observed_count)
    return float((model_score - target) ** 2)

model_score = np.log(30)
matching_error = squared_glove_residual(30, model_score)
different_error = squared_glove_residual(3, model_score)

print("log targets:", [round(float(np.log(c)), 3) for c in (3, 30, 300)])
print("matching count error:", round(matching_error, 3))
print("different count error:", round(different_error, 3))

Output

log targets: [1.099, 3.401, 5.704]
matching count error: 0.0
different count error: 5.302

Word2Vec and GloVe give each known word one static row. That is efficient, but it creates two problems: new spellings have no row, and ambiguous words have only one.

Compose New Spellings from Pieces

Support systems constantly encounter variants: reship, reshipped, reshipping, or tracking codes mixed into prose. FastText represents a word using character n-grams as well as its whole-word identity, letting related spellings share parameters.^[4]

FastText subword composition showing a boundary-marked word split into character n-grams, hashed into buckets, and summed with the full-word vector. — Character pieces provide a usable starting vector for an unseen variant such as reshipped.

07-share-character-ngrams.py

def character_ngrams(word, n=3):
    marked = f"<{word}>"
    return {marked[i:i + n] for i in range(len(marked) - n + 1)}

known = character_ngrams("reship")
new_form = character_ngrams("reshipped")
shared = sorted(known & new_form)

print("shared pieces:", shared)
print("new spelling can reuse pieces:", len(shared) > 0)

Output

shared pieces: ['<re', 'esh', 'hip', 'res', 'shi']
new spelling can reuse pieces: True

Subword composition helps with unfamiliar surface forms. It still doesn't decide which meaning an existing ambiguous token carries.

One Static Row Can't Choose a Sense

Read these messages:

text

"Dispute the charge on order 8142."
"Charge the scanner before the warehouse shift."

The word charge is spelled the same way in both. A static embedding lookup returns the same vector, even though the first case belongs near billing language and the second near device language.

08-expose-static-sense-collision.py

import numpy as np

static = {
    "charge": np.array([0.8, 0.2]),
    "refund": np.array([0.2, 1.0]),
    "scanner": np.array([1.0, 0.1]),
}

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

billing_charge = static["charge"]
device_charge = static["charge"]

print("same charge vector:", np.array_equal(billing_charge, device_charge))
print("billing similarity to refund:", round(cosine(billing_charge, static["refund"]), 3))
print("device similarity to refund:", round(cosine(device_charge, static["refund"]), 3))

Output

same charge vector: True
billing similarity to refund: 0.428
device similarity to refund: 0.428

Because the two charge vectors are identical, every downstream comparison begins from the same mistaken compromise. The representation needs evidence from the sentence.

Let Tokens Exchange Context

ELMo showed that a token's representation can be produced from a bidirectional language model and selected from multiple learned layers, rather than stored as one context-free vector.^[5]

ELMo architecture showing character CNN token representations feeding separate forward and backward LSTM stacks, then a learned weighted layer mixture. — ELMo's output depends on sentence context because each directional language model contributes to the token state.

BERT later trained a Transformer encoder with masked language modeling: hide some tokens, then predict them using context on both sides. For a visible token state, encoder self-attention can incorporate evidence before and after that token.^[6]

BERT encoder stack showing token, segment, and position embeddings entering bidirectional transformer layers that produce contextual token representations. — The visibility mask shows available context positions; it deliberately doesn't pretend to show learned attention weights.

You don't need a pretrained model to see the central mechanism. In this toy contextualizer, the same starting vector for charge is mixed with a clue from its sentence.

09-mix-context-into-charge.py

import numpy as np

charge_start = np.array([0.8, 0.2])
clues = {
    "dispute": np.array([0.0, 1.0]),  # billing evidence
    "scanner": np.array([1.0, 0.0]),  # device evidence
}

def mix_with_context(token_vector, clue_vector):
    return 0.5 * token_vector + 0.5 * clue_vector

billing_state = mix_with_context(charge_start, clues["dispute"])
device_state = mix_with_context(charge_start, clues["scanner"])

print("billing charge state:", billing_state.tolist())
print("device charge state:", device_state.tolist())
print("same final state:", np.array_equal(billing_state, device_state))

Output

billing charge state: [0.4, 0.6]
device charge state: [0.9, 0.1]
same final state: False

The weights above are illustrative, not learned model parameters. A trained attention layer learns which context tokens should contribute, and later layers can refine that state again.

Bidirectional and Causal Context

Not every contextual representation may look to the same places.

A BERT-style encoder token can use tokens on both sides of its position during encoding.
A GPT-style causal language model predicts the next token from the prefix, so a token state can't use later tokens while preserving that next-token objective. The original GPT work used a Transformer decoder for generative pretraining.^[7]

For the first token in charge the scanner, the right-hand clue scanner is visible to an encoder but not to a causal decoder state at position charge.

10-compare-visibility-masks.py

import numpy as np

tokens = ["charge", "the", "scanner"]
encoder_visibility = np.ones((len(tokens), len(tokens)), dtype=int)
causal_visibility = np.tril(np.ones((len(tokens), len(tokens)), dtype=int))

def visible_tokens(mask, position):
    return [token for token, visible in zip(tokens, mask[position]) if visible]

print("encoder state for charge sees:", visible_tokens(encoder_visibility, 0))
print("causal state for charge sees:", visible_tokens(causal_visibility, 0))

Output

encoder state for charge sees: ['charge', 'the', 'scanner']
causal state for charge sees: ['charge']

Both designs produce contextual states. Their visibility rules suit different training objectives and later tasks.

Timeline from count-based static vectors through prediction and subword methods to sentence-dependent encoder and causal states. — The progression is about information available to each token state, not a blanket ranking of model families.

Similarity Needs a Geometry Check

Embedding applications often use cosine similarity: a value near 1 means two vectors point in similar directions. That measurement is useful only if the space behaves sensibly.

Research on contextual representations found that token vectors can be anisotropic: many vectors share a dominant direction, inflating raw cosine similarities even for tokens that shouldn't be treated as alike.^[8]

Embedding geometry comparison showing an isotropic point cloud spread across directions versus an anisotropic cone where contextual vectors cluster around dominant directions. — When many vectors share one direction, raw cosine similarity can hide the distinction a task needs.

This tiny example has a large shared horizontal component. Raw cosine says the two states are almost identical; subtracting their shared mean reveals that their informative vertical components oppose one another.

11-diagnose-shared-direction.py

import numpy as np

billing = np.array([10.0, 1.0])
device = np.array([10.0, -1.0])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

raw_similarity = cosine(billing, device)
mean_direction = (billing + device) / 2
centered_similarity = cosine(billing - mean_direction, device - mean_direction)

print("raw cosine:", round(raw_similarity, 3))
print("centered cosine:", round(centered_similarity, 3))

Output

raw cosine: 0.98
centered cosine: -1.0

Centering isn't a universal production recipe. It is a diagnostic reminder: measure retrieval or classification quality on held-out examples instead of trusting a similarity score in isolation.

Choose What the Failure Requires

No representation is best by slogan. Start with the failure mode and the measurement that would prove improvement.

Need	Useful starting point	What to measure
Small fixed vocabulary, simple classifier	Trainable embedding lookup	Held-out classification quality and latency
Rare spelling variants such as `reshipped`	Character or subword-aware static vectors	Recall on unseen variants
Ambiguous words such as billing/device `charge`	Contextual token states	Accuracy on sense-dependent cases
Search over whole support messages	Sentence or chunk embedding model	Retrieval precision and recall on real queries

Contextual token states aren't automatically good message-level retrieval vectors. Pooling, task training, and evaluation still matter. You will build those choices in later retrieval lessons.

What You Built

You now have a concrete chain from token IDs to context-dependent meaning:

An embedding table maps each token ID to one starting vector.
Co-occurrence, Word2Vec, and GloVe learn static geometry from usage evidence.
FastText-style character pieces let related spellings share parameters.
Static vectors fail when identical token text carries different senses.
Context mixing gives each occurrence its own state.
Cosine similarity still requires evaluation because geometry can be distorted.

Mastery Check

Evaluation rubric

Foundational: You can explain why an embedding lookup returns one stored row for every occurrence of a token ID.
Intermediate: You can build a co-occurrence or prediction example and show why the two meanings of charge require context mixing.
Advanced: You can choose a representation for a measured failure case and test whether cosine geometry supports the chosen metric.

Follow-up questions

Common pitfalls

Treating token IDs as meaning: An ID only selects a row; training and context create useful geometry.
Calling subword handling disambiguation: Character pieces help unfamiliar spellings, not multiple senses of one familiar word.
Trusting cosine without an evaluation set: A high score can come from shared directions rather than task-relevant similarity.

Next Step

Continue to Perplexity & Model Evaluation

Embeddings decide what information a language model can carry forward. Next you'll measure how well its predicted token probabilities fit real text.

PreviousBPE, WordPiece, and SentencePiece

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Indexing by Latent Semantic Analysis

Deerwester, S., et al. · 1990 · JASIS

Efficient Estimation of Word Representations in Vector Space.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. · 2013 · arXiv preprint

GloVe: Global Vectors for Word Representation.

Pennington, J., Socher, R., & Manning, C. D. · 2014

Enriching Word Vectors with Subword Information.

Bojanowski, P., et al. · 2017

Deep contextualized word representations.

Peters, M., et al. · 2018 · NAACL 2018

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Devlin, J., et al. · 2019 · NAACL 2019

Improving Language Understanding by Generative Pre-Training.

Radford, A., et al. · 2018

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.

Ethayarajh, K. · 2019

Back to Topics

LearnCore LLM FoundationsStatic to Contextual Embeddings

📝MediumNLP Fundamentals

Static to Contextual Embeddings

Turn token IDs into vectors, learn what nearby usage captures, and see why a word such as charge needs sentence-dependent representations.

13 min read

Learning path

Step 47 of 155 in the full curriculum

BPE, WordPiece, and SentencePiece Perplexity & Model Evaluation

A tokenizer can turn a support request into IDs:

text

"dispute the charge" -> [91, 12, 407]
"charge the scanner"  -> [407, 12, 982]

IDs solve naming. They don't tell a model that refund resembles return, or that charge means a billing event in one message and supplying power in another.

An embedding gives each token a of numbers. A contextual representation goes one step further: it adjusts that vector using the sentence around this particular occurrence.

By the end of this lesson, you'll build both ideas from scratch and watch the same leave with two different meanings.

Look Up Coordinates for Token IDs

Suppose a tokenizer has a vocabulary of size V, and you choose embedding dimension d. The model stores an embedding E with shape V x d.

Row E[token_id] is the starting vector for that token.
Training changes rows so tokens used in similar situations become useful to the prediction task.
Before context mixing, every occurrence of the same token ID receives the same row.

That final point matters. If charge is ID 0, both billing and device sentences initially retrieve row E[0].

01-look-up-token-rows.py

import numpy as np

vocab = {"charge": 0, "refund": 1, "scanner": 2}
embedding_table = np.array([
    [0.80, 0.20],  # charge
    [0.15, 0.95],  # refund
    [0.92, 0.10],  # scanner
])

billing_charge = embedding_table[vocab["charge"]]
device_charge = embedding_table[vocab["charge"]]

print("billing start:", billing_charge.tolist())
print("device start:", device_charge.tolist())
print("same starting row:", np.array_equal(billing_charge, device_charge))

Output

billing start: [0.8, 0.2]
device start: [0.8, 0.2]
same starting row: True

A lookup table alone can't distinguish senses. First, though, it has to learn a useful map of token usage.

Learn Meaning from Neighboring Words

The simplest evidence is a co-occurrence count: for every token, count words within a fixed context window.

02-count-context-neighbors.py

from collections import Counter, defaultdict

messages = [
    "refund approved send return label".split(),
    "replacement approved send return label".split(),
    "refund request needs label".split(),
    "tracking update from carrier hub".split(),
    "shipment delayed at carrier hub".split(),
]

window = 2
neighbors = defaultdict(Counter)

for message in messages:
    for center_index, center in enumerate(message):
        left = max(0, center_index - window)
        right = min(len(message), center_index + window + 1)
        for context_index in range(left, right):
            if context_index != center_index:
                neighbors[center][message[context_index]] += 1

print("refund context:", neighbors["refund"].most_common(4))
print("replacement context:", neighbors["replacement"].most_common(4))
print("carrier context:", neighbors["carrier"].most_common(4))

Output

refund context: [('approved', 1), ('send', 1), ('request', 1), ('needs', 1)]
replacement context: [('approved', 1), ('send', 1)]
carrier context: [('hub', 2), ('update', 1), ('from', 1), ('delayed', 1)]

03-compress-ppmi-with-svd.py

import numpy as np

words = ["refund", "replacement", "carrier", "tracking"]
contexts = ["label", "approved", "hub", "delayed"]
counts = np.array([
    [8, 6, 0, 0],  # refund
    [7, 6, 0, 0],  # replacement
    [0, 0, 8, 5],  # carrier
    [0, 0, 7, 6],  # tracking
], dtype=float)

expected = counts.sum(axis=1, keepdims=True) @ counts.sum(axis=0, keepdims=True) / counts.sum()
ppmi = np.maximum(np.log((counts + 1e-9) / (expected + 1e-9)), 0.0)
u, singular_values, _ = np.linalg.svd(ppmi, full_matrices=False)
vectors = u[:, :2] * np.sqrt(singular_values[:2])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print("refund vs replacement:", round(cosine(vectors[0], vectors[1]), 3))
print("refund vs carrier:", round(cosine(vectors[0], vectors[2]), 3))

Output

refund vs replacement: 1.0
refund vs carrier: 0.0

This tiny dataset is designed to make the pattern visible. Real corpora have far more contexts, imperfect wording, and competing meanings.

Predict Neighbors with Word2Vec

Counting first and compressing later isn't the only way to learn a static embedding table. Word2Vec trains vectors through prediction:

CBOW predicts a center token from its surrounding tokens.
Skip-gram predicts surrounding tokens from a center token.

Both objectives reward vectors that help predict observed neighborhoods. Mikolov and colleagues introduced these two architectures in 2013.^[2]

For Skip-gram, a training set can be built directly from windows. With window size 2, the center arrived produces one positive pair for each visible neighbor.

04-create-skipgram-pairs.py

tokens = "package arrived at carrier hub today".split()
window = 2
pairs = []

for center_index, center in enumerate(tokens):
    left = max(0, center_index - window)
    right = min(len(tokens), center_index + window + 1)
    for context_index in range(left, right):
        if context_index != center_index:
            pairs.append((center, tokens[context_index]))

arrived_pairs = [pair for pair in pairs if pair[0] == "arrived"]
print(arrived_pairs)

Output

[('arrived', 'package'), ('arrived', 'at'), ('arrived', 'carrier')]

05-compare-negative-sampling-loss.py

import numpy as np

center = np.array([1.0, 0.0])
observed_neighbor = np.array([1.2, 0.1])
wrong_neighbor = np.array([-1.0, 0.1])
negative_samples = [np.array([-0.9, -0.2]), np.array([-1.1, 0.0])]

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

def negative_sampling_loss(center_vector, positive_vector, negatives):
    positive_loss = -np.log(sigmoid(center_vector @ positive_vector))
    negative_loss = sum(-np.log(sigmoid(-(center_vector @ n))) for n in negatives)
    return float(positive_loss + negative_loss)

observed_loss = negative_sampling_loss(center, observed_neighbor, negative_samples)
wrong_loss = negative_sampling_loss(center, wrong_neighbor, negative_samples)

print("observed pair loss:", round(observed_loss, 3))
print("wrong pair loss:", round(wrong_loss, 3))
print("training prefers observed pair:", observed_loss < wrong_loss)

Output

observed pair loss: 0.892
wrong pair loss: 1.942
training prefers observed pair: True

No analogy trick is required to understand the useful result: after enough examples, tokens that support similar neighbor predictions can end up near one another.

Fit Global Counts with GloVe

06-fit-glove-log-count.py

import numpy as np

def squared_glove_residual(observed_count, model_score):
    target = np.log(observed_count)
    return float((model_score - target) ** 2)

model_score = np.log(30)
matching_error = squared_glove_residual(30, model_score)
different_error = squared_glove_residual(3, model_score)

print("log targets:", [round(float(np.log(c)), 3) for c in (3, 30, 300)])
print("matching count error:", round(matching_error, 3))
print("different count error:", round(different_error, 3))

Output

log targets: [1.099, 3.401, 5.704]
matching count error: 0.0
different count error: 5.302

Word2Vec and GloVe give each known word one static row. That is efficient, but it creates two problems: new spellings have no row, and ambiguous words have only one.

Compose New Spellings from Pieces

07-share-character-ngrams.py

def character_ngrams(word, n=3):
    marked = f"<{word}>"
    return {marked[i:i + n] for i in range(len(marked) - n + 1)}

known = character_ngrams("reship")
new_form = character_ngrams("reshipped")
shared = sorted(known & new_form)

print("shared pieces:", shared)
print("new spelling can reuse pieces:", len(shared) > 0)

Output

shared pieces: ['<re', 'esh', 'hip', 'res', 'shi']
new spelling can reuse pieces: True

Subword composition helps with unfamiliar surface forms. It still doesn't decide which meaning an existing ambiguous token carries.

One Static Row Can't Choose a Sense

Read these messages:

text

"Dispute the charge on order 8142."
"Charge the scanner before the warehouse shift."

The word charge is spelled the same way in both. A static embedding lookup returns the same vector, even though the first case belongs near billing language and the second near device language.

08-expose-static-sense-collision.py

import numpy as np

static = {
    "charge": np.array([0.8, 0.2]),
    "refund": np.array([0.2, 1.0]),
    "scanner": np.array([1.0, 0.1]),
}

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

billing_charge = static["charge"]
device_charge = static["charge"]

print("same charge vector:", np.array_equal(billing_charge, device_charge))
print("billing similarity to refund:", round(cosine(billing_charge, static["refund"]), 3))
print("device similarity to refund:", round(cosine(device_charge, static["refund"]), 3))

Output

same charge vector: True
billing similarity to refund: 0.428
device similarity to refund: 0.428

Because the two charge vectors are identical, every downstream comparison begins from the same mistaken compromise. The representation needs evidence from the sentence.

Let Tokens Exchange Context

ELMo showed that a token's representation can be produced from a bidirectional language model and selected from multiple learned layers, rather than stored as one context-free vector.^[5]

You don't need a pretrained model to see the central mechanism. In this toy contextualizer, the same starting vector for charge is mixed with a clue from its sentence.

09-mix-context-into-charge.py

import numpy as np

charge_start = np.array([0.8, 0.2])
clues = {
    "dispute": np.array([0.0, 1.0]),  # billing evidence
    "scanner": np.array([1.0, 0.0]),  # device evidence
}

def mix_with_context(token_vector, clue_vector):
    return 0.5 * token_vector + 0.5 * clue_vector

billing_state = mix_with_context(charge_start, clues["dispute"])
device_state = mix_with_context(charge_start, clues["scanner"])

print("billing charge state:", billing_state.tolist())
print("device charge state:", device_state.tolist())
print("same final state:", np.array_equal(billing_state, device_state))

Output

billing charge state: [0.4, 0.6]
device charge state: [0.9, 0.1]
same final state: False

The weights above are illustrative, not learned model parameters. A trained attention layer learns which context tokens should contribute, and later layers can refine that state again.

Bidirectional and Causal Context

Not every contextual representation may look to the same places.

A BERT-style encoder token can use tokens on both sides of its position during encoding.
A GPT-style causal language model predicts the next token from the prefix, so a token state can't use later tokens while preserving that next-token objective. The original GPT work used a Transformer decoder for generative pretraining.^[7]

For the first token in charge the scanner, the right-hand clue scanner is visible to an encoder but not to a causal decoder state at position charge.

10-compare-visibility-masks.py

import numpy as np

tokens = ["charge", "the", "scanner"]
encoder_visibility = np.ones((len(tokens), len(tokens)), dtype=int)
causal_visibility = np.tril(np.ones((len(tokens), len(tokens)), dtype=int))

def visible_tokens(mask, position):
    return [token for token, visible in zip(tokens, mask[position]) if visible]

print("encoder state for charge sees:", visible_tokens(encoder_visibility, 0))
print("causal state for charge sees:", visible_tokens(causal_visibility, 0))

Output

encoder state for charge sees: ['charge', 'the', 'scanner']
causal state for charge sees: ['charge']

Both designs produce contextual states. Their visibility rules suit different training objectives and later tasks.

Similarity Needs a Geometry Check

Embedding applications often use cosine similarity: a value near 1 means two vectors point in similar directions. That measurement is useful only if the space behaves sensibly.

11-diagnose-shared-direction.py

import numpy as np

billing = np.array([10.0, 1.0])
device = np.array([10.0, -1.0])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

raw_similarity = cosine(billing, device)
mean_direction = (billing + device) / 2
centered_similarity = cosine(billing - mean_direction, device - mean_direction)

print("raw cosine:", round(raw_similarity, 3))
print("centered cosine:", round(centered_similarity, 3))

Output

raw cosine: 0.98
centered cosine: -1.0

Centering isn't a universal production recipe. It is a diagnostic reminder: measure retrieval or classification quality on held-out examples instead of trusting a similarity score in isolation.

Choose What the Failure Requires

No representation is best by slogan. Start with the failure mode and the measurement that would prove improvement.

Need	Useful starting point	What to measure
Small fixed vocabulary, simple classifier	Trainable embedding lookup	Held-out classification quality and latency
Rare spelling variants such as `reshipped`	Character or subword-aware static vectors	Recall on unseen variants
Ambiguous words such as billing/device `charge`	Contextual token states	Accuracy on sense-dependent cases
Search over whole support messages	Sentence or chunk embedding model	Retrieval precision and recall on real queries

Contextual token states aren't automatically good message-level retrieval vectors. Pooling, task training, and evaluation still matter. You will build those choices in later retrieval lessons.

What You Built

You now have a concrete chain from token IDs to context-dependent meaning:

An embedding table maps each token ID to one starting vector.
Co-occurrence, Word2Vec, and GloVe learn static geometry from usage evidence.
FastText-style character pieces let related spellings share parameters.
Static vectors fail when identical token text carries different senses.
Context mixing gives each occurrence its own state.
Cosine similarity still requires evaluation because geometry can be distorted.

Mastery Check

Evaluation rubric

Foundational: You can explain why an embedding lookup returns one stored row for every occurrence of a token ID.
Intermediate: You can build a co-occurrence or prediction example and show why the two meanings of charge require context mixing.
Advanced: You can choose a representation for a measured failure case and test whether cosine geometry supports the chosen metric.

Follow-up questions

Common pitfalls

Treating token IDs as meaning: An ID only selects a row; training and context create useful geometry.
Calling subword handling disambiguation: Character pieces help unfamiliar spellings, not multiple senses of one familiar word.
Trusting cosine without an evaluation set: A high score can come from shared directions rather than task-relevant similarity.

Next Step

Continue to Perplexity & Model Evaluation

Embeddings decide what information a language model can carry forward. Next you'll measure how well its predicted token probabilities fit real text.

PreviousBPE, WordPiece, and SentencePiece

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Indexing by Latent Semantic Analysis

Deerwester, S., et al. · 1990 · JASIS

Efficient Estimation of Word Representations in Vector Space.

Mikolov, T., Chen, K., Corrado, G., & Dean, J. · 2013 · arXiv preprint

GloVe: Global Vectors for Word Representation.

Pennington, J., Socher, R., & Manning, C. D. · 2014

Enriching Word Vectors with Subword Information.

Bojanowski, P., et al. · 2017

Deep contextualized word representations.

Peters, M., et al. · 2018 · NAACL 2018

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Devlin, J., et al. · 2019 · NAACL 2019

Improving Language Understanding by Generative Pre-Training.

Radford, A., et al. · 2018

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.

Ethayarajh, K. · 2019

Static to Contextual Embeddings

Look Up Coordinates for Token IDs

Learn Meaning from Neighboring Words

Predict Neighbors with Word2Vec

Fit Global Counts with GloVe

Compose New Spellings from Pieces

One Static Row Can't Choose a Sense

Let Tokens Exchange Context

Bidirectional and Causal Context

Similarity Needs a Geometry Check

Choose What the Failure Requires

What You Built

Mastery Check

Evaluation rubric

Follow-up questions

Why can't a static embedding table represent billing charge and device charge differently in these two sentences?

What does a FastText-style character n-gram representation fix for reshipped, and what does it not fix for charge?

Why should a retrieval team test cosine similarity on held-out tickets before trusting contextual vectors?

Common pitfalls

Static to Contextual Embeddings

Look Up Coordinates for Token IDs

Learn Meaning from Neighboring Words

Predict Neighbors with Word2Vec

Fit Global Counts with GloVe

Compose New Spellings from Pieces

One Static Row Can't Choose a Sense

Let Tokens Exchange Context

Bidirectional and Causal Context

Similarity Needs a Geometry Check

Choose What the Failure Requires

What You Built

Mastery Check

Evaluation rubric

Follow-up questions

Why can't a static embedding table represent billing charge and device charge differently in these two sentences?

What does a FastText-style character n-gram representation fix for reshipped, and what does it not fix for charge?

Why should a retrieval team test cosine similarity on held-out tickets before trusting contextual vectors?

Common pitfalls