Sentence Embeddings & Contrastive Loss

In the production-agent capstone, document_qa_v2 found return-policy-us-v3 before the agent drafted an answer about a cracked tablet. That contract deliberately hid one important mechanism: how did a policy passage become a good candidate for the question in the first place?

A sentence embedding maps a query or passage to one fixed-width vector. A retriever can then find passages near a customer question even when wording changes:

This chapter teaches how contrastive learning shapes that vector space. The agent still needs evidence and authorization gates; embeddings decide which text gets considered before those gates run.

From word embeddings to sentence embeddings

Word embeddings gave individual tokens numerical coordinates. Retrieval needs one vector for an entire query or policy passage. The challenge is to compress variable-length text into a fixed-size representation whose neighborhood ordering is useful for the task.

Averaging context-free word vectors is a useful baseline, but it loses order and context. Don't confuse that baseline with mean pooling contextual token outputs inside a trained sentence encoder: pooling specifies how to produce one vector; the training objective determines whether its geometry supports retrieval. A common objective is contrastive learning, which rewards a relevant pair for scoring above irrelevant candidates.

---

Naive approaches (and why they fail)

Mean pooling of word embeddings

A simple approach to creating a sentence embedding is to calculate the word vectors for each token in the sentence and then average them. This naive baseline is fast but loses important structural information. The function below uses a tiny local vector table so you can run the baseline and see exactly what it throws away:

The final line is the problem: both sentences produce the same vector because averaging ignores order.

Problems

• Ignores word order: "carrier delayed order" and "order delayed carrier" have identical embeddings because the sum of vectors is commutative ($A+B = B+A$), even though word order can change what the sentence emphasizes.
• Common-word dilution: Frequent words and boilerplate phrasing can wash out the signal from rare, informative tokens.
• No context: Polysemous words like "charge" (payment dispute vs. powering a scanner) get averaged into a single messy vector.

[CLS] token from BERT

Using the [CLS] (Classification) token from BERT (Bidirectional Encoder Representations from Transformers) directly as a sentence embedding isn't a sound retrieval baseline. Reimers & Gurevych (2019) showed that BERT's paired-input architecture is impractical for large semantic search and introduced SBERT so independently encoded sentence vectors could be compared with cosine similarity.^[1] A pretrained task token hasn't been trained to make nearest-neighbor distance a relevance score.

The anisotropy problem

Contextual representations can show anisotropy: vectors occupy a narrow cone in the high-dimensional space rather than using directions evenly.^[2] Think of many policy passages all pointing toward one generic "support text" direction. Their cosine scores can look high even when they answer different questions. Contrastive objectives can improve this geometry by rewarding aligned positives while penalizing competing candidates.

---

Contrastive learning for sentence embeddings

The core idea

The goal of contrastive learning is simple but powerful: reshape the embedding space so that sentences with similar meaning land close to each other, while unrelated sentences are pushed apart.

Before we formalize it, recall what "close" means here. Cosine similarity measures how aligned two vectors point (their angle), ignoring their length. For two unit vectors (length 1), it's simply their dot product. A value of +1 means same direction, 0 means orthogonal directions, and a negative value means opposing directions. None of those numbers proves semantic identity or irrelevance by itself; only an evaluated embedding model makes cosine ranking useful. The next lesson studies this scoring rule in detail.

With that in mind, contrastive training teaches the model to:

• Pull embeddings of similar sentences toward each other (high cosine similarity)
• Push embeddings of dissimilar sentences away (low cosine similarity)

SimCSE (Simple Contrastive Learning of Sentence Embeddings)^[3] demonstrated an unusually small self-supervised construction: pass the same sentence through the encoder twice while dropout supplies different noisy views, then treat those views as a positive pair. Wang & Isola's analysis gives useful vocabulary for the resulting geometry: alignment asks whether positives are close, and uniformity asks whether representations avoid crowding into a small region of the hypersphere.^[4] E5 later trained single-vector text embeddings contrastively from a large weakly supervised pair corpus.^[5]

The most common training objective here is Information Noise Contrastive Estimation (InfoNCE): for one anchor sentence, the model should rank the true match above every other candidate in the batch.

InfoNCE loss

Think of contrastive learning as matching a support ticket to its true duplicate while pushing away unrelated tickets. You want the matching pair close together and every non-match farther away. InfoNCE (Information Noise Contrastive Estimation) mathematically formalizes this push-and-pull dynamic.

Before looking at the formula, walk through a tiny concrete case. Suppose you have a batch of 2 query-passage pairs, and after normalizing their embeddings you measure cosine similarities (which, for unit vectors, are just dot products):

For query $q_1$, the true match is $p_1$ (similarity 0.90). The other passage in the batch, $p_2$, acts as an in-batch negative (similarity 0.20). InfoNCE wants the model to make $p_1$ look more likely than $p_2$.

For this worked row, choose a sharp temperature $\tau = 0.05$ and compute the loss contribution for $q_1$ step by step:

1. Scale the similarities: positive logit = $0.90 / 0.05 = 18.0$, negative logit = $0.20 / 0.05 = 4.0$
2. Exponentiate (this turns logits into unnormalized probabilities): $\exp(18.0) \approx 65{,}659{,}969$, $\exp(4.0) \approx 54.6$
3. Normalize into a probability for the positive: $65{,}659{,}969 / (65{,}659{,}969 + 54.6) \approx 0.99999917$
4. Take negative log: $-\log(0.99999917) \approx 0.00000083$ (tiny loss; the model is already very confident)

If the model were wrong ($q_1$ similarity to $p_1$ only 0.20, to $p_2$ 0.90), the positive probability would drop to about $8.3 \times 10^{-7}$ and the loss would jump to roughly 14 nats. The optimizer would receive a strong gradient pushing the correct pair closer.

The standard contrastive loss for a batch of $N$ positive pairs:^[6]

$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\text{sim}(z_i, z_i^+) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(z_i, z_j^+) / \tau)}$

Reading the formula

For each example $i$, compare its similarity to the true match $z_i^+$ (numerator) against the full batch-level denominator. That denominator includes the positive pair itself plus every other candidate in the batch. Temperature $\tau$ controls how sharply the model distinguishes between similar and dissimilar pairs. The loss says: "make the true pair more likely than every alternative in the batch."

Where:

• $z_i, z_i^+$ are embeddings of a positive pair (e.g., query and relevant document)
• $\tau$ is the temperature parameter
• All non-matching examples in that denominator act as in-batch negatives

The following copy-runnable implementation shows the same calculation without hiding the matrix math behind a framework. Production training code would vectorize this in PyTorch, JAX, or another tensor library, but the loop below makes the denominator explicit:

The equation is often expanded into raw exponentials when calculating a small example on paper. Code should compute the same expression with log-sum-exp or a framework cross-entropy operation, so large logits don't overflow.

Triplet loss

A second contrastive objective, triplet loss, focuses on individual anchor-positive-negative triplets instead of comparing one anchor against a whole candidate pool:

$\mathcal{L} = \max(0, d(a, p) - d(a, n) + m)$

Where:

• $d(\cdot, \cdot)$ is the Euclidean distance between embeddings
• $m$ is a margin hyperparameter
• $a$ is the anchor, $p$ is the positive, $n$ is the negative

The loss enforces that the anchor must be closer to the positive than to the negative by at least margin $m$: $d(a, p) + m \leq d(a, n)$. If this constraint is already satisfied, the loss is zero. The margin prevents the model from wasting capacity pushing already-distant negatives even farther away.

Worked example: computing triplet loss by hand

Consider three sentences about return labels:

After encoding, suppose the distances are:

• $d(a, p) = 0.2$ (the paraphrase is close)
• $d(a, n) = 0.5$ (the hard negative is farther, but not by much)

With margin $m = 0.1$, plug into the formula:

$0.2 - 0.5 + 0.1 = -0.2$

$\max(0, -0.2) = 0$

The loss is zero because the model already satisfies the margin constraint: the positive is closer than the negative by more than 0.1. Now imagine a bad model where $d(a, p) = 0.5$ and $d(a, n) = 0.3$ (the negative is closer than the positive):

$0.5 - 0.3 + 0.1 = 0.3$

$\max(0, 0.3) = 0.3$

A non-zero loss tells the optimizer to push the anchor and positive together while pushing the negative away until the gap exceeds the margin.

Key differences from InfoNCE

• Triplet loss compares a chosen negative against a margin, so mining determines most of its learning signal.
• InfoNCE compares each anchor with a candidate pool. Batches supply negatives cheaply, but they can also contain false negatives.
• Neither objective is an automatic win. Choose data construction deliberately and evaluate held-out retrieval failures, not training loss alone.

Temperature parameter τ

Temperature controls the "sharpness" of the softmax distribution over similarity scores:

For the worked similarity gap, $0.90 - 0.20 = 0.70$, changing temperature changes the positive probability:

There is no universal best temperature. Tune it against held-out retrieval failures and implement the loss stably. Low temperature amplifies mislabeled or false negatives; overflow is an implementation bug that stable log-softmax or cross-entropy avoids.

---

Training strategies for sentence embeddings

Training a good embedding model requires good training data. You need pairs of sentences that are either semantically similar or different, and you need enough diversity to teach the model real distinctions. Two main approaches dominate.

Supervised: fine-tuning on NLI data

Natural Language Inference (NLI) labels whether a hypothesis follows from a premise (entailment), conflicts with it (contradiction), or does neither (neutral). Entailment is directional, not a promise that two texts are interchangeable.

SBERT trained Siamese and triplet architectures with NLI supervision and evaluated sentence similarity behavior.^[1] Supervised SimCSE uses entailment as a positive and the corresponding contradiction as a hard negative.^[3] This is a useful training construction, but you still need retrieval evaluation before treating two policy passages as substitutes.

SBERT's architectural move is simple but important: it runs both sentences through the same encoder with shared weights, pools each sentence into a single vector, and then trains on top of those pooled embeddings.^[1] During inference, you only keep the single-sentence encoding path. That shared-weight Siamese network setup (two inputs processed through the same shared encoder) is what makes precomputing document embeddings and doing nearest-neighbor search practical.

Self-supervised: SimCSE

What if you don't have labeled NLI data? SimCSE (Simple Contrastive Learning of Sentence Embeddings) from Gao et al. (2021)^[3] shows you don't need it. The trick is elegant: pass the same sentence through the encoder twice with different dropout masks. Since dropout randomly zeros out different neurons each time, you get two slightly different embedding vectors for the same sentence. These two views are positives (they should be similar), while all other sentences in the batch are negatives.

This gives surprisingly strong embeddings without labeled pairs, although supervised SimCSE performs better across the paper's reported STS tasks.^[3] The key insight is that the stochasticity in the transformer forward pass, which is normally just a regularization technique, becomes an implicit data augmentation mechanism for contrastive learning.

Data augmentation

Beyond dropout, augmentations are hypotheses about meaning preservation:

• Dropout masks (SimCSE): different mask patterns per forward pass
• Verified paraphrases or back-translations: use only after checking that policy scope and exceptions survive
• Domain pairs: mine resolved duplicate questions that cite the same approved policy passage
• Re-ranking with cross-encoders: score candidate paraphrases before accepting them as positives

Avoid casual word deletion or insertion for policy text: dropping "not," a product category, or an approval condition changes the rule while incorrectly labeling the pair positive.

In-batch negatives in practice

Most contrastive learning implementations use in-batch negatives by default: for a batch of $N$ positive pairs, each anchor has one matching positive, and the other $N-1$ candidates in the batch act as negatives. This is efficient because you get many negatives "for free" without explicitly labeling them.

Larger batches increase the chance of informative competitors, but they also increase the chance of false negatives: another row may cite the same relevant policy as the anchor while the loss treats it as wrong. Distributed training commonly gathers embeddings across GPUs before computing this loss. Plain gradient accumulation doesn't create more in-batch negatives unless the implementation explicitly reuses embeddings across microbatches.

---

Hard negative mining

Why hard negatives matter

Random negatives are too easy. The model quickly learns to distinguish "refund label missing" from "warehouse forklift battery." Hard negatives force the model to learn subtle semantic distinctions, which is where real retrieval quality comes from.

Mining strategies

1. In-batch negatives

Use other examples in the batch. Simple, scales with batch size, but negatives may be too easy.

2. BM25 negatives

Use a lexical search algorithm like BM25 (Best Matching 25, a sparse retrieval function based on keyword frequency) to find documents that are lexically similar but semantically different:

3. Cross-encoder mining

Use a cross-encoder to find high-BM25-score but low-relevance passages. The code below demonstrates the control flow with deterministic helper functions: lexical overlap stands in for BM25, and a small relevance scorer stands in for the cross-encoder. The important behavior is the filter: keep candidates that look lexically relevant but fail the semantic relevance check.

4. Iterative mining

Re-mine hard negatives after each training epoch using the improved model. This progressively finds harder examples as the model improves.

---

Bi-encoder vs cross-encoder

Bi-encoder (dual encoder)

Encode query and document independently, compare with dot product or cosine similarity:

Advantages

Documents can be pre-encoded and indexed. At query time, you only encode the query once, then use an ANN (Approximate Nearest Neighbor) index to retrieve candidates sublinearly in practice.

Disadvantages

No cross-attention between query and document; it may miss fine-grained relevance signals.

Cross-encoder

Concatenate query and document, process jointly through full transformer:

Advantages

Full attention can model phrase order, negation, and query-document interactions that a single-vector score misses. On a suitable shortlist, this often improves precision over first-stage vector scoring.

Disadvantages

You must run inference for every (query, document) pair. If you scored the full corpus directly, that's O(N) transformer passes per query, which is too slow for large-scale search.

Late interaction: ColBERT

ColBERT (Contextualized Late Interaction over BERT)^[7] offers a middle ground between bi-encoder and cross-encoder:

Instead of a single embedding per document, ColBERT stores per-token embeddings and computes relevance using MaxSim: for each query token, find the maximum similarity to any document token, then sum.

Advantages

Retains token-level matching signals while documents can still be pre-encoded.

Disadvantages

Much larger index size (one vector per token instead of per document).

This architecture choice is a budget tradeoff, not a universal ranking. A bi-encoder protects first-stage recall at corpus scale; a cross-encoder can improve precision for a bounded candidate set; late interaction spends more index space to preserve token-level evidence.

A production reranking pattern

In a production system, these two architectures are combined in a two-stage pipeline to balance speed and accuracy. The function below illustrates this pattern with deterministic scores: first retrieve a broad candidate set using a fast bi-encoder score, then re-score those candidates with a slower cross-encoder score and return the final ranking.

---

Instruction-tuned embeddings

The problem with task ambiguity

"Label" means different things for clustering (group return-label documents), retrieval (find barcode-label troubleshooting), and classification (is this about shipping or inventory?). Traditional embeddings produce the same vector regardless of the downstream task. This limits performance when a single model must serve multiple purposes.

Task-specific prefixes and instructions

Some embedding families expose task prefixes or lightweight instructions that steer the encoder toward retrieval, clustering, or classification. E5 is a simple example: it distinguishes inputs like query: and passage: during contrastive pre-training.^[5] INSTRUCTOR-style models go further and condition the embedding on an explicit task instruction.^[8] The format is model-specific. Prefixes that help one family can hurt another, so follow its training or model documentation.

This example keeps the families separate on purpose. The exact prefix or instruction string depends on the embedding family you chose:

This doesn't mean one magic prefix solves every task. It means some embedding families expect an extra conditioning signal. Use the format documented for that specific model family, then benchmark it on your own retrieval, clustering, or classification workload.

---

Matryoshka representation learning (MRL)

The idea

Matryoshka embeddings are named after nesting dolls because selected prefix widths are trained to remain useful on their own. For example, a full 768-dimensional embedding can be trained together with 128- and 32-dimensional prefixes. You then choose among trained and evaluated widths based on the storage-quality budget.

Train embeddings so that selected prefix widths preserve useful representations under their own losses:^[9]

For a contrastively trained embedding model, the loss can be computed at several predefined truncation points simultaneously. The following runnable toy implementation slices full embeddings down to smaller prefixes, calculates the same InfoNCE objective at each prefix, and averages the losses:

Why it matters

---

Evaluation: STS and MTEB

Semantic textual similarity (STS)

Before large-scale benchmarks like MTEB existed, a common benchmark for evaluating sentence embeddings was Semantic Textual Similarity (STS). Datasets like STS-B (STS Benchmark) provide sentence pairs rated by human annotators for semantic relatedness:

To evaluate a model, you compute the cosine similarity for every pair using the model's embeddings, and then calculate the Spearman rank correlation between the model's similarity scores and the human ratings. A high correlation means the model's embedding space aligns well with human judgment.

MTEB (Massive text embedding benchmark)

As models improved, optimizing only for STS was no longer sufficient. An embedding model excellent at STS might fail miserably at information retrieval or clustering. To address this, the Massive Text Embedding Benchmark (MTEB) was introduced.^[10] The original MTEB paper evaluated models on 58 datasets grouped into 8 task categories, giving a much broader view of embedding quality than STS alone:

Choosing a model in practice

Don't anchor on a single MTEB average. Deployment success usually depends on a few operational questions:

• Does the model expect plain text, query/passage prefixes, or explicit instructions?
• Can you shorten the embedding width safely, or are you locked into the full dimensionality?
• How well does it handle your language mix, domain jargon, and query length distribution?
• What are the latency, throughput, and memory costs once you batch and index it at production scale?
• Do you still need BM25 or a cross-encoder reranker to hit Recall@K and NDCG targets?

In practice, architecture often matters more than a tiny leaderboard delta. A strong bi-encoder with good hard negatives, sensible chunking, and a reranker usually beats blindly swapping to the latest model name.

The original MTEB finding is the durable lesson: no one model dominated every task category.^[10] Evaluate policy retrieval, reranking, languages, latency, and storage behavior that match your deployment instead of selecting by one aggregate score.

Carry the evidence boundary into retrieval evaluation

document_qa_v2 can use an embedding retriever to propose policy passages, but vector proximity isn't authorization. A release test should verify both retrieval quality and that unapproved text never becomes answer evidence:

Key libraries and tools

Building embedding-based systems requires the right tooling:

---

Mastery check

Key concepts

• alignment and uniformity in embedding space
• InfoNCE numerator, denominator, and temperature
• hard negatives vs easy negatives
• bi-encoder vs cross-encoder vs late interaction
• reranking as recall first, then precision
• Matryoshka prefix training for safe dimension cuts

Evaluation rubric

• Foundational: Derives the InfoNCE objective and explains what the numerator, denominator, and temperature do.
• Intermediate: Explains why raw BERT [CLS] embeddings fail for semantic search without sentence-level contrastive fine-tuning.
• Intermediate: Explains why hard negatives matter more than random negatives once the model already separates broad topics.
• Advanced: Compares bi-encoders, cross-encoders, and late-interaction models by latency, indexability, accuracy, and storage.
• Advanced: Explains ColBERT's MaxSim scoring and why it keeps more token-level signal than a single document vector.
• Advanced: Explains Matryoshka embeddings and when shorter prefixes are worth the storage-accuracy tradeoff.
• Advanced: Designs a two-stage production retrieval pipeline with recall and latency budgets defended quantitatively.

Follow-up questions

Common pitfalls

Raw [CLS] is treated like a search-ready sentence embedding

Symptom: Nearly every query-document pair gets suspiciously similar cosine scores. Cause: Raw BERT [CLS] vectors weren't tuned for semantic retrieval and can inherit poorly discriminative anisotropic geometry. Fix: Start from a sentence embedding model or fine-tune with a contrastive objective before building nearest-neighbor search.

Negatives stay too easy

Symptom: Training loss falls, but recall on realistic queries barely moves. Cause: Random negatives stop teaching once the model separates unrelated topics. Fix: Mine BM25 or cross-encoder negatives that share words with the anchor but answer a different intent.

The reranker is asked to save missing recall

Symptom: The reranker looks good in pairwise inspection, yet the right document is often absent in production results. Cause: The correct passage never entered the shortlist. Fix: Tune first-stage Recall@K separately, then widen candidate budget before blaming the reranker.

Dimensions are shortened blindly

Symptom: Index storage drops as expected, but retrieval quality falls off a cliff. Cause: A standard embedding vector was truncated without prefix-aware training or provider support. Fix: Use Matryoshka-trained or provider-documented shortening controls and benchmark each target width.

Task conditioning is ignored

Symptom: One embedding model works for clustering but underperforms on retrieval. Cause: The model family expected query/passage prefixes or instructions, but every input was embedded as plain text. Fix: Follow the model card format for retrieval, clustering, and classification separately.

---

Try it yourself

These exercises let you verify your understanding without needing a GPU cluster.

Exercise 1: compute triplet loss by hand

Given an anchor $a$, positive $p$, and negative $n$ with distances $d(a,p) = 0.3$ and $d(a,n) = 0.7$, compute the triplet loss for margins $m = 0.1$ and $m = 0.5$. In which case does the model still have work to do?

Solution sketch: For $m = 0.1$: $0.3 - 0.7 + 0.1 = -0.3$, so $\max(0, -0.3) = 0$. The margin is already satisfied. For $m = 0.5$: $0.3 - 0.7 + 0.5 = 0.1$, so $\max(0, 0.1) = 0.1$. The larger margin forces the model to pull the positive even closer or push the negative farther away.

Exercise 2: spot the embedding mistake

A teammate reports that their semantic search system returns nearly identical similarity scores for every query-document pair. They are using a pretrained BERT model and taking the [CLS] token as the sentence embedding. What is the most likely cause, and what is the smallest change that would fix it?

Solution sketch: Raw BERT [CLS] embeddings weren't trained to make cosine distance a semantic-retrieval score, and anisotropic geometry can make their scores poorly discriminative. The smallest fix is to switch to a sentence embedding model that was fine-tuned with a sentence-level objective (for example, SBERT or E5), rather than using raw BERT.

Exercise 3: design a two-stage retrieval pipeline

You have 2 million support tickets and a latency budget of 200 ms per query. You own a bi-encoder that encodes a query in 10 ms and a cross-encoder that scores one query-document pair in 15 ms. Why is scoring the full corpus with the cross-encoder impossible, and what pipeline would hit the latency budget?

Solution sketch: $2{,}000{,}000 \times 15\,\text{ms} = 30{,}000{,}000\,\text{ms} \approx 8.3$ hours per query. The cross-encoder is far too slow for the full corpus. Reserve 10 ms for query encoding and choose a top-10 shortlist only if ANN lookup and overhead fit inside the remaining 40 ms: reranking then costs $10 \times 15\,\text{ms} = 150\,\text{ms}$, for at most 200 ms total. If Recall@10 is inadequate, the budget or reranker throughput must change; silently widening to top 100 violates the requirement.

---

What you have now

You now understand why raw transformer outputs fail for semantic search, how contrastive learning reshapes the embedding space, and how to train and deploy sentence embeddings in production. You can explain InfoNCE and triplet loss with concrete numbers, mine hard negatives from lexical overlap, and design a two-stage retrieval pipeline that balances speed and accuracy.

The next article, Embedding Similarity & Quantization, builds directly on this foundation. You will learn the mathematical details of cosine similarity versus dot product, how Matryoshka truncation changes the accuracy-storage tradeoff, and how to quantize embeddings to 8-bit, 4-bit, or binary formats for large-scale indexes. Those techniques only make sense once you understand why the embedding space was shaped by contrastive loss in the first place.