Word to Contextual Embeddings

How does a computer understand that "king" and "queen" are related, or that "Paris" is to "France" what "Tokyo" is to "Japan"? It can't read a dictionary. It needs numbers. The solution is to give every word a set of coordinates, like pinning cities on a map. Words with similar meanings end up close together, and the directions between them encode relationships. These numerical coordinates are called embeddings, and they are the foundation of how every language model (from ChatGPT to Gemini) understands meaning.

This article traces the complete arc: from early counting methods through Word2Vec's breakthrough to today's context-aware representations in BERT and GPT, where the same word gets different coordinates depending on its surroundings.

🎯 Core concept: Understanding embeddings means grasping the core idea behind representation learning: that a word's meaning comes from the company it keeps, that static coordinates evolved into dynamic ones, and the practical tradeoffs involved. This knowledge is foundational to every modern NLP and LLM system. (See also: how text becomes tokens before it becomes embeddings, and sentence embeddings for scaling this idea to whole passages.)

---

The distributional hypothesis: the foundation of everything

Before any algorithm, understand the core insight that makes all word embeddings possible:

"You shall know a word by the company it keeps." (J.R. Firth, 1957)

Think of it this way: if you moved to a new city and saw a store you'd never heard of, you could guess what it sells by looking at its neighbors. A store between a bakery and a deli is probably a food shop. A store between a nail salon and a hair salon is probably a beauty shop. You didn't need to go inside. The neighborhood told you everything.

Words work the same way. The word "espresso" tends to appear near "latte," "barista," and "café," so a computer can figure out it's a coffee-related word just by tracking those patterns. This is the distributional hypothesis: words that appear in similar contexts have similar meanings. Every embedding method (from TF-IDF to GPT) is at its core an operationalization of this idea.

text
1Context: "The ___ sat on the mat"
2
3Words that fit: cat, dog, child, kitten, puppy
4→ These words should have similar embeddings
5
6Words that don't fit: democracy, algorithm, quantum
7→ These should have very different embeddings

The evolution of embeddings is the story of increasingly powerful ways to capture this distributional information:

---

1. Count-based methods: the starting point

Before neural approaches, words were represented using co-occurrence statistics.

TF-IDF (Term Frequency–Inverse Document Frequency) weights words by importance within a document relative to a corpus. Simple but effective for information retrieval, it's still used today in search engines.

LSA (Latent Semantic Analysis), introduced by Deerwester et al. (1990)^[1], applies Singular Value Decomposition (SVD) to the term-document matrix to discover latent semantic dimensions.

Key limitation: These methods produce a single vector per word regardless of context. The word "bank" has the same representation whether it means a financial institution or a riverbank. This is the polysemy problem, and solving it would drive the next decade of research.

---

2. Word2Vec: the neural embedding revolution

Mikolov et al. (2013)^[2] introduced two neural architectures that learn dense word vectors by predicting words from their local context. This was the breakthrough that launched modern NLP embeddings.

Two architectures

CBOW (Continuous Bag of Words) predicts the center word from surrounding context:

$P(w_t \mid w_{t-c}, \ldots, w_{t-1}, w_{t+1}, \ldots, w_{t+c})$

Reading the formula: given the surrounding words (the "context window" of size $c$ on each side), predict the word in the middle. For example, given "the ___ sat on", predict "cat."

Skip-gram predicts context words from the center word (inverse of CBOW):

$P(w_{t+j} \mid w_t) \quad \text{for } j \in [-c, c], j \neq 0$

Reading the formula: this is the reverse. Given the center word "cat", predict which words likely surround it ("the", "sat", "on"). Skip-gram tends to work better for rare words because each word gets more training updates.

💡 Key distinction: CBOW is faster and better for frequent words. Skip-gram is slower but captures rare words better (each rare word generates multiple training examples as a center word)^[2].

Training: negative sampling

Here's the problem: to learn good embeddings, the model needs to check its prediction against every word in the vocabulary, often 100,000+ words. That's like grading a multiple-choice test with 100,000 answer options for every single question.

Negative sampling (Mikolov et al., 2013)^[2] is an elegant shortcut. Instead of comparing against all 100K words, the model picks a handful of random "wrong" answers (negatives) and just learns to tell the difference between the real context word and those few imposters. It's like a flashcard game: "Is 'cat' a real neighbor of 'sat'?" (yes ✅) "Is 'democracy' a real neighbor of 'sat'?" (no ❌). This tiny binary quiz is dramatically cheaper while still teaching the model which words belong together. The code below demonstrates how to compute this negative sampling loss for a single word pair. It takes the embeddings of the center word, the actual context word, and a set of random negative samples, returning a scalar loss value that forces the true pair to have high similarity and the fake pairs to have low similarity:

python
1import numpy as np
2
3def skip_gram_loss(center_vec: np.ndarray, 
4                   context_vec: np.ndarray, 
5                   neg_samples: np.ndarray) -> float:
6    """
7    Computes the negative sampling loss for a single skip-gram pair.
8    
9    Args:
10        center_vec: Embedding of center word (d,)
11        context_vec: Embedding of context word (d,)
12        neg_samples: Embeddings of k negative samples (k, d)
13    
14    Returns:
15        Scalar loss value
16    """
17    # 1. Positive pair: maximize log probability of real context word
18    # sigmoid(u_o^T v_c)
19    pos_score = np.dot(context_vec, center_vec)
20    # sigmoid(x) = 1 / (1 + exp(-x))
21    pos_prob = 1 / (1 + np.exp(-pos_score))
22    pos_loss = -np.log(pos_prob + 1e-10)
23    
24    # 2. Negative pairs: maximize log probability of NOT being context words
25    # sigmoid(-u_k^T v_c) for all k negative samples
26    neg_scores = np.dot(neg_samples, center_vec) 
27    # We want these to be low, so maximize 1-sigmoid(x) = sigmoid(-x)
28    neg_prob_inv = 1 / (1 + np.exp(neg_scores)) 
29    neg_loss = -np.sum(np.log(neg_prob_inv + 1e-10))
30    
31    return pos_loss + neg_loss

Negative words are sampled proportional to $f(w)^{3/4}$, where $f(w)$ is the word frequency. The $3/4$ exponent gives rare words slightly more chance of being selected as negatives, improving their representations.

The famous analogy property

Word2Vec's most striking property is that semantic relationships are captured as linear directions in the embedding space:

text
1king - man + woman ≈ queen
2Paris - France + Italy ≈ Rome  
3walked - walking + swimming ≈ swam

This works because regularities in the training data create consistent vector offsets. The "gender direction" (man → woman) is roughly the same vector regardless of the word pair.

⚠️ Reality check: Research by Ethayarajh et al. (2019)^[3] and others has shown that the king–queen analogy is somewhat cherry-picked. Many analogies don't work cleanly, and the cosine similarity nearest neighbor often isn't the "correct" answer. Understanding this nuance distinguishes deep expertise from surface-level knowledge.

Practical details

---

3. GloVe: global vectors

Pennington, Socher & Manning (2014)^[4] at Stanford combined the best of count-based and prediction-based methods.

Core insight: co-occurrence ratios encode meaning

Here's a clever detective trick that GloVe exploits. Say you're trying to figure out the difference between "ice" and "steam." Looking at which words appear near each one individually isn't very helpful, since both appear near "water." But if you look at the ratio, the pattern jumps out: the word "solid" appears 100× more often with "ice" than with "steam," while "gas" appears 100× more with "steam." Neutral words like "water" appear equally with both, giving a ratio near 1. It's like comparing two suspects by looking at who they hang out with relative to each other.

The key insight is that ratios of co-occurrence probabilities, not raw probabilities, capture semantic relationships:

GloVe optimizes word vectors so their dot product equals the log co-occurrence count:

$J = \sum_{i,j=1}^{V} f(X_{ij}) \left( \mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2$

Reading the formula: for every pair of words in the vocabulary, the loss penalizes the difference between (a) the dot product of their learned vectors and (b) the log of how often they actually co-occur in the corpus. The weighting function $f(X_{ij})$ prevents extremely common pairs (like "the, of") from overwhelming the training. The result: vectors whose geometry directly encodes co-occurrence statistics.

Where:

• •$X_{ij}$ = co-occurrence count of words $i$ and $j$
• •$f(x)$ = weighting function that caps at $x_{\text{max}} = 100$ (prevents frequent pairs from dominating)
• •$\mathbf{w}_i, \tilde{\mathbf{w}}_j$ = word and context vectors

GloVe vs Word2Vec

💡 Insight: Levy & Goldberg (2014) showed that Word2Vec with negative sampling is implicitly factorizing a shifted PMI matrix. The methods are more similar than they appear^[5].

---

4. FastText: solving the out-of-vocabulary problem

Bojanowski et al. (2017)^[6] at Facebook extended Word2Vec by representing words as bags of character n-grams:

text
1Word: "where" (with n=3..6)
2N-grams: {"<wh", "whe", "her", "ere", "re>", "<whe", "wher", "here", "ere>", 
3           "<wher", "where", "here>", "<where", "where>", "<where>"}
4
5Embedding("where") = Σ embedding(ngram) for each ngram

The killer feature: FastText can compute embeddings for OOV (Out-of-Vocabulary) words, words it has never seen, by summing the vectors of their subword n-grams. This is critical for:

• •Morphologically rich languages (Turkish, Finnish, Hungarian): "evlerimizdekilerden" → meaningful embedding from subparts
• •Typos and misspellings: "recieve" gets a reasonable embedding from shared n-grams with "receive"
• •Neologisms and slang: novel words get useful representations

🎯 Still useful today: FastText embeddings remain the go-to for resource-constrained settings, feature engineering for tabular ML, and simple similarity search where a full transformer is overkill.

---

5. ELMo: the contextual revolution

Every method so far has a fundamental flaw: the word "bank" always gets the same set of coordinates, whether you're talking about a river bank or a savings bank. It's like giving every person named "Jordan" the same ID photo, ignoring everything about who they actually are.

Peters et al. (2018)^[7] changed this with Embeddings from Language Models (ELMo), the first widely successful contextual word representations. Now words are like chameleons: they change color based on their surroundings. This was the inflection point, the end of the "one vector per word" era.

Architecture

ELMo uses a 2-layer bidirectional LSTM (Long Short-Term Memory) trained as a language model:

Layer specialization

Each layer captures different linguistic information:

The final representation is a task-specific weighted combination of all layers:

$\text{ELMo}_k = \gamma \sum_{j=0}^{L} s_j \cdot \mathbf{h}_{k,j}$

Reading the formula: the final representation for token $k$ is a weighted blend of all layers' outputs. The weights $s_j$ are learned during fine-tuning, so the model can decide "for sentiment analysis, I need more of the upper (semantic) layer; for POS tagging, more of the lower (syntactic) layer." The scaling factor $\gamma$ adjusts the overall magnitude.

The polysemy breakthrough

text
1"The river bank was steep and muddy."
2   → ELMo("bank") ≈ [terrain, geography, nature...]
3
4"I need to visit the bank to deposit a check."
5   → ELMo("bank") ≈ [finance, money, institution...]

For the first time, the same word gets different vectors in different contexts. This single insight drove massive improvements across NLP benchmarks.

Limitation: The bidirectional LSTMs process left and right contexts independently, then concatenate. They don't jointly attend to both directions simultaneously. BERT would solve this.

---

6. BERT: truly bidirectional transformers

Devlin et al. (2019)^[8] replaced LSTMs with Transformers and introduced truly bidirectional pre-training, establishing the "pre-train then fine-tune" workflow.

Pre-training objectives

Masked Language Modeling (MLM): Randomly mask 15% of tokens and predict them:

• •80% replaced with [MASK]
• •10% replaced with a random token (prevents the model from only learning to fill masks)
• •10% kept unchanged (brings representations closer to fine-tuning, where no masks exist)

text
1Input:  "The cat [MASK] on the mat"
2Target: "sat"

Next Sentence Prediction (NSP): Binary classification: are two sentences consecutive? (Later work showed NSP's contribution is minimal. RoBERTa dropped it entirely and improved results.)

Architecture

Why bidirectional self-attention matters

Unlike ELMo's concatenated left/right contexts, BERT's self-attention jointly conditions on the full context in every layer:

text
1"I accessed my bank account"
2→ Attention: "bank" attends to BOTH "accessed" AND "account"
3→ Result: financial meaning
4
5"The river bank was steep"
6→ Attention: "bank" attends to BOTH "river" AND "steep"  
7→ Result: geographical meaning

This is fundamentally more powerful than ELMo's approach of processing left and right independently, then concatenating.

💡 Analogy, reading a mystery novel: ELMo is like two detectives investigating a crime scene separately. One reads the clues left-to-right, the other right-to-left, and they compare notes at the end. BERT is like a single detective who can look at all the clues simultaneously, spotting connections the two separate investigators would miss. When "bank" appears between "river" and "account," BERT sees both context words interacting with each other through "bank." ELMo never gets that cross-directional evidence.

The fine-tuning approach

BERT established a workflow that dominated NLP for years:

1. •Pre-train on massive unlabeled corpus (BooksCorpus + Wikipedia, ~3.3B words)
2. •Add a task-specific head (classification layer, span extraction, etc.)
3. •Fine-tune all parameters on labeled downstream data
4. •Result: State-of-the-art on 11 NLP tasks simultaneously at release

---

7. GPT: autoregressive representations at scale

Radford et al. (2018, 2019)^[9] at OpenAI took the opposite approach: unidirectional (left-to-right) transformer pre-training.

The autoregressive objective

$\mathcal{L} = -\sum_{t=1}^{T} \log P(x_t \mid x_1, \ldots, x_{t-1}; \theta)$

Reading the formula: for each position in the text, the model tries to predict the next word using only the words that came before it. The loss sums up how wrong those predictions were (via negative log-probability). Better predictions = lower loss. This "predict the next word" game is all GPT needs to learn language.

BERT vs GPT: the key tradeoff

Why GPT's approach won at scale

Despite BERT's bidirectional advantage for understanding tasks, GPT's autoregressive approach scaled better:

1. •Unified framework: Any text task can be cast as text generation (no task-specific heads needed)
2. •In-context learning: Few-shot examples in the prompt replace fine-tuning entirely (Brown et al., 2020)^[10]
3. •Emergent abilities: Larger models unlock new capabilities like chain-of-thought reasoning, code generation, and tool use (Wei et al., 2022)^[11]
4. •Natural generation: Autoregressive models naturally produce coherent multi-sentence output

💡 Technical nuance: Saying "BERT is better because it's bidirectional" is incomplete. The correct nuance is: BERT has stronger per-token representations for understanding tasks, but GPT's autoregressive formulation enables generation and scaling properties that proved more valuable for general-purpose AI.

---

8. Embedding geometry: what research tells us

A frequently overlooked but critical concept is the geometry of embeddings: how they're distributed in high-dimensional space.

Anisotropy: the cone problem

Imagine you're in a dark room with a flashlight, and you scatter glow-in-the-dark balls everywhere. Ideally, they'd be spread evenly across the room so you could tell any two balls apart by their positions. But what if all the balls ended up clustered in the narrow beam of the flashlight? Suddenly, every ball looks like it's in roughly the same direction, and it's hard to distinguish them.

Ethayarajh (2019)^[3] at Stanford discovered that this is exactly what happens with embeddings. In BERT, ELMo, and GPT-2, word embeddings are anisotropic: they occupy a narrow cone in the embedding space rather than being uniformly distributed.

Key findings:

• •On average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding, meaning contextual representations are highly context-dependent
• •Upper layers produce more context-specific representations than lower layers
• •Lower-layer BERT representations, when reduced to their first principal component, actually outperform GloVe and FastText on standard static embedding benchmarks

🔬 Why this matters: Anisotropy affects how cosine similarity behaves. Two random words might have surprisingly high cosine similarity simply because all vectors point in roughly the same direction, not because the words are semantically related. This motivates techniques like whitening and isotropy calibration in production systems.

---

The complete evolution at a glance

---

When to use what (practical guide)

A strong engineer knows when not to use the latest model:

---

Key takeaways

• •Trace the evolution: Understand the progression from count-based (statistics) → prediction-based (local context) → contextual (full sentence awareness).
• •Technical distinctions matter: Know the difference between Word2Vec's local window, GloVe's global matrix factorization, and BERT's bidirectional attention.
• •Context is king: ELMo proved that one vector per word is insufficient; BERT proved that bidirectional attention beats concatenated LSTMs.
• •Scale wins: GPT showed that autoregressive objectives, while locally less powerful than bidirectional ones, scale better to general reasoning tasks.
• •Geometry awareness: Remember that embedding spaces are often anisotropic (cone-shaped), which distorts cosine similarity.

Common misconceptions

• •"BERT is always better." False. Static embeddings are faster, cheaper, and sufficient for many simple similarity or feature engineering tasks.
• •"ELMo and BERT are both bidirectional in the same way." False. ELMo concatenates independent left/right LSTMs; BERT jointly attends to all tokens.
• •"GPT is strictly worse than BERT for understanding." False. While BERT is more sample-efficient for understanding, scaled-up GPT models perform well on NLU tasks via few-shot prompting.
• •"King - Man + Woman = Queen always works." False. This is a cherry-picked example. Many analogies fail, and the geometry is often distorted.
• •"High dimensionality is always better." False. Diminishing returns set in (often around 300d for static, 768d+ for contextual), and higher dimensions increase compute cost.

Summary

1. •Start with the problem: Words need numerical representations for neural networks to process them.
2. •Milestones: Count-based → Word2Vec → GloVe → FastText → ELMo → BERT → GPT.
3. •Key insights: Co-occurrence → prediction → subwords → context → attention → scale.
4. •Modern state: Today's LLMs use contextual embeddings computed by transformer layers, but static embeddings remain a vital tool for efficiency.

---