Embedding Similarity & Quantization

api-doc-retriever-v1 has a strict boundary: a retriever may propose policy passages, but only approved evidence may support an answer. As the approved policy collection grows, every query has to search it quickly without silently degrading which passage is retrieved.

Suppose a user asks whether a contractor can open production dashboards. The correct result is the approved access policy, not a private draft note that happens to repeat "contractor dashboard" many times. Both passages become long lists of numbers called embeddings (or vectors). Retrieval compares those vectors, but its scoring and compression choices determine whether the approved policy even reaches the evidence gate.

Start with three vector scoring rules, see exactly when they rank results identically, and then tackle memory pressure. You'll compress stored vectors with scalar, product, and binary quantization, measure what each shortcut loses, and build the shortlist-and-rescore pattern that protects answer quality.

Three ways to measure closeness

An embedding is just a point in high-dimensional space. If you have two points, you can ask how near they are. There are three common rulers, and each answers a slightly different question.

• Cosine similarity asks: "Do they point in the same direction?"
• Dot product asks: "Do they point in the same direction, and how large are their magnitudes?"
• Euclidean distance asks: "What's the straight-line distance between them?"

For any pipeline that normalizes every vector to length 1, cosine similarity and dot product become mathematically equivalent for ranking. That turns the metric decision into a systems decision: use the scoring rule your search library can execute quickly, then keep the same normalization contract at ingest time and query time.

A tiny concrete example shows how the two rulers differ without needing a computer.

A worked example in two dimensions

Suppose we embed two incident notes into tiny 2D vectors so we can draw them on paper:

• Note A: "GPU cache warmed" maps to [3, 4]
• Note B: "Cache warm on GPU" maps to [4, 3]

Both point northeast, so they should be similar. Measure that similarity with each ruler.

Cosine similarity

Cosine similarity ignores length and looks only at direction. The formula is:

$$\cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}$$

First, compute the dot product in the numerator. Multiply matching dimensions and add:

$$\mathbf{A} \cdot \mathbf{B} = (3 \times 4) + (4 \times 3) = 12 + 12 = 24$$

Next, compute the lengths (norms) of each vector. The length of a vector is the square root of the sum of squared components:

$$\|\mathbf{A}\| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$$ $$\|\mathbf{B}\| = \sqrt{4^2 + 3^2} = \sqrt{16 + 9} = \sqrt{25} = 5$$

Now divide:

$$\cos(\theta) = \frac{24}{5 \times 5} = \frac{24}{25} = 0.96$$

A score of 0.96 means these invented vectors point in nearly the same direction. Whether direction corresponds to relevance in a real index is something the embedding training and retrieval evaluation must establish.

Dot product

The dot product skips the normalization step and uses the raw sum we already calculated:

$$\mathbf{A} \cdot \mathbf{B} = 24$$

Because both vectors happen to have the same length (5), the dot product gives a ranking that matches cosine similarity. But if one vector were much longer, the dot product would change even though the angle stayed the same.

Euclidean distance

Subtract the dimensions, square the differences, sum them, and take the square root:

$$\|\mathbf{A} - \mathbf{B}\|_2 = \sqrt{(3-4)^2 + (4-3)^2} = \sqrt{1 + 1} = \sqrt{2} \approx 1.41$$

A small Euclidean distance also tells us these notes are close in space. For this single pair, all three rulers describe a close relationship. Add candidates with different norms, even in two dimensions, and their rankings can diverge. The choice of ruler changes which notes you retrieve.

The same arithmetic becomes a few lines of NumPy. This is a direct way to test whether your retrieval code matches the math you just worked by hand:

If vector A is [3, 4] and vector C is [30, 40], what happens to cosine similarity and dot product when you compare both vectors to [4, 3]?

Answer

Cosine similarity stays the same because A and C point in the same direction. Dot product grows by 10x because C is 10x longer. That is the core difference: cosine ignores magnitude, while dot product preserves it.

Cosine similarity: direction only

Cosine similarity is useful when the retrieval contract says vector length shouldn't change ranking. It compares direction, not "meaning" directly. A strong embedding model can arrange relevant policy text in similar directions, but cosine alone can't prove relevance or authorization.

Separate direction from signal strength. Cosine checks whether two arrows point toward the same concept cluster; it ignores how long the arrows are. Raw dot product cares about both alignment and vector length. Which signal matters is set by model training and validated on retrieval queries, not chosen by intuition after indexing.

Cosine similarity ranges from $-1$ to $1$: 1 means same direction, 0 means orthogonal, and -1 means opposite direction. Its key property is magnitude invariance. A vector of [1, 1] has the same cosine direction as [100, 100] because they point the same way.

When to use cosine similarity

• When the model card or your retrieval evaluation calls for angular comparison.
• When document and query vectors are normalized consistently and vector norm isn't intended to carry ranking signal.
• When an experiment on your own queries confirms that angular scoring retrieves the right evidence.

Dot product: direction plus magnitude

The dot product is an un-normalized alignment score between two vectors:

$$\mathbf{A} \cdot \mathbf{B} = \sum_{i=1}^n a_i b_i$$

Multiply each pair of matching dimensions and add them all up. Unlike cosine similarity, there's no dividing by lengths, so if one vector is twice as long, the dot product doubles. That makes it the cheaper query-time metric once vectors have already been unit-normalized, but it also makes raw dot products sensitive to how "big" each embedding is.

The dot product ranges from $-\infty$ to $+\infty$. Its key property is sensitivity to both direction and magnitude.

When to use dot product

• When magnitude is intentionally part of the learned score, as in maximum inner product search (MIPS).
• When your training setup or ANN library is optimized for inner-product search.
• When vectors are already L2-normalized, because inner product and cosine produce the same ranking.
• When both indexed vectors and query vectors are unit-normalized before scoring, so inner product preserves cosine ranking without doing norm divisions during candidate scoring.

When all three metrics agree

Some APIs return unit-length embeddings, and many production pipelines normalize vectors at ingest time even when the model itself doesn't. Treat normalization as an explicit contract between model output, index construction, and query serving. Don't assume it's implicit because the model is modern.

When both vectors are unit length ($\|\mathbf{A}\| = \|\mathbf{B}\| = 1$), the mathematical relationship between our similarity metrics collapses into a clean equivalence. The denominator in the cosine similarity formula becomes $1 \times 1 = 1$, making it identical to the raw dot product. Euclidean distance also becomes a direct function of the dot product:

$$\cos(\mathbf{A}, \mathbf{B}) = \mathbf{A} \cdot \mathbf{B} = 1 - \frac{\|\mathbf{A} - \mathbf{B}\|^2}{2}$$

Why this matters

When all vectors have length 1, cosine similarity, dot product, and squared Euclidean distance all induce the same ranking. In practice, pick the metric your library optimizes best and keep query-time and index-time conventions consistent. After the stored vectors and each incoming query have been normalized once, inner product maps directly to fast matrix multiplication kernels without norm divisions during candidate scoring.

This retrieval fixture makes the contract visible. A private-note fixture has a deliberately large raw norm and wins a raw dot-product search. Once all candidates and the query are normalized, the approved policy wins under both cosine and unit-vector dot product.

You shouldn't infer that every large-norm vector is bad. The safe conclusion is narrower: if the serving contract is cosine-like ranking, skipping normalization changes what the index is optimizing.

Euclidean distance: straight-line gap

Euclidean distance (also called L2 distance) measures the straight-line distance between two points:

$$\|\mathbf{A} - \mathbf{B}\|_2 = \sqrt{\sum_{i=1}^n (a_i - b_i)^2}$$

Subtract matching dimensions, square each difference, sum them up, and take the square root. That is the straight-line distance between two points in high-dimensional space, just like measuring distance on a map, but with hundreds of dimensions instead of two.

High dimension doesn't make Euclidean distance automatically wrong. On unit-normalized vectors, squared Euclidean distance produces exactly the same ranking as cosine and dot product through the equation above. Without that normalization, L2 distance also includes norm differences. Use it when the model, index, and evaluation are built around that geometry.

Euclidean distance ranges from $0$ to $+\infty$, where 0 means identical points. It also appears inside product quantization, where the original PQ formulation approximates squared L2 distances between a full-precision query and compressed database vectors.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11}

Choosing how many dimensions you need

Embedding dimensionality controls how much semantic information the vector can encode. That capacity comes with direct storage, memory-bandwidth, and latency costs. Choosing dimensions is a real architecture decision, not a leaderboard footnote.

Width is a model-and-evaluation contract

A dimension count has no universal quality meaning by itself. A 512-dimensional model trained well for your support queries can outperform a different 1,536-dimensional model. Width comparisons become defensible when you hold the model family, scoring convention, corpus, and evaluation set fixed.

Dimension count and normalization are separate choices. A supported shorter vector still needs the serving normalization contract documented by its model or applied by your pipeline.

Why dimensions matter

For a fixed representation family, more stored dimensions have deterministic systems costs:

1. Storage: Each float32 is 4 bytes. 1M vectors at 1536d is about 5.7 GiB of raw payload.
2. Exact scoring work: A dot product or L2 comparison does linear work in the number of dimensions, written $O(d)$, per candidate.
3. Memory bandwidth: Loading vectors from memory is often the bottleneck for Approximate Nearest Neighbor (ANN) search, a class of algorithms that find close vectors without an exhaustive search.
4. End-to-end latency: Halving dimensions doesn't promise halved latency because graph traversal, filters, networking, and reranking remain. Benchmark before paying or saving the memory bill.

Matryoshka embeddings

Some embedding families are trained so selected truncated prefixes stay useful. Matryoshka Representation Learning (MRL) optimizes losses at multiple chosen representation sizes during training.^{[2]Reference 2Matryoshka Representation Learning.https://arxiv.org/abs/2205.13147} Hosted APIs can expose a supported shortening control too; OpenAI's embeddings guide documents a dimensions parameter for text-embedding-3 models.^{[3]Reference 3Vector embeddingshttps://developers.openai.com/api/docs/guides/embeddings}

Matryoshka embeddings are named after nesting dolls because one representation contains selected shorter prefixes. That doesn't guarantee every arbitrary cut point, every task, or every query keeps acceptable recall. Use widths that the training setup or provider supports, then evaluate each proposed serving width on your own evidence queries.

Standard embedding training minimizes a single loss function (like contrastive loss) on the full vector. MRL minimizes a weighted sum of losses at a selected set of truncation points, written as $d \in \mathcal{D}$. This nested structure encourages the trained prefix widths to form useful representations on their own.

MRL is different from post-hoc dimensionality reduction such as PCA. PCA projects already-trained embeddings into a smaller linear subspace, usually by preserving variance on a fixed dataset. MRL changes training itself so prefix slices are useful without a separate projection step. PCA can still be useful for frozen embeddings, but MRL is cleaner when the model was trained for truncation and you want one serving path with several dimension budgets.

Supported prefix shortening can reduce raw payload while keeping one model family in the serving stack. OpenAI's embeddings documentation, for example, lists default sizes of 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large, and documents shorter embeddings through the dimensions parameter.^{[3]Reference 3Vector embeddingshttps://developers.openai.com/api/docs/guides/embeddings} Those documented capabilities still don't replace evaluation on your retrieval corpus.

If you ask a provider for a shorter embedding through the API, follow that provider's normalization contract. Manual slicing is different. If you cut a prefix from an already generated vector, normalize that sliced prefix before indexing. The local check enforces that contract without any external API call:

Normalization is necessary for a unit-vector scoring contract, but it can't restore information removed by truncation. This deliberately small fixture makes that distinction concrete: the last two dimensions separate an approved policy from a private draft note, so the shortened representation loses one correct result.

This isn't a benchmark for a real MRL model. It's the release test shape you need: evaluate supported widths on approved-query fixtures, including near-duplicate but unauthorized passages.

You shorten a unit-normalized 3072-dimensional embedding by taking only the first 256 values. Should you index that prefix immediately?

Answer

No. Slicing usually changes the vector's L2 norm, so normalize the 256-dimensional prefix before indexing. Otherwise cosine, dot product, and Euclidean conventions no longer match the index contract you think you are using.

The memory problem: a sizing example

Work through this step by step. The same arithmetic appears whenever you size a vector database.

Scenario: You have 1 million embeddings from a hosted API, each with 1,536 dimensions, stored as 32-bit floats (FP32). How much RAM do you need for the raw vector payload, and how can you reduce it?

Step 1: Calculate the baseline

Each dimension is a float32, which takes 4 bytes.

$$1{,}000{,}000 \times 1{,}536 \times 4 \text{ bytes} \approx 6.14 \text{ GB}$$

That's just the raw numbers. Real indexes add overhead for graph links, metadata, and replicas, but the vector payload alone is already over 6 GB.

Step 2: Apply 8-bit scalar quantization

If you map each float32 to a single byte (INT8), you cut storage by roughly 4x:

$$1{,}000{,}000 \times 1{,}536 \times 1 \text{ byte} \approx 1.54 \text{ GB}$$

Step 3: Apply product quantization aggressively

Suppose you split each 1,536-dimensional vector into 96 subvectors of 16 dimensions each, and you represent each subvector with an 8-bit centroid ID. The compressed vector is just 96 bytes:

$$1{,}000{,}000 \times 96 \text{ bytes} \approx 96 \text{ MB}$$

That's a 64x compression from the original FP32 payload. You're now comparing approximate codes instead of exact floats, so you typically pair PQ with a higher-precision reranking stage.

The byte math is easy to turn into a sizing helper. Notice the difference between decimal gigabytes, which vendors often use in pricing copy, and binary gibibytes, which operating systems usually report:

Re-embedding cost scales with total input tokens, not document count alone. Compute it as total_tokens / 1_000_000 * embedding_price_per_1M_tokens, using the current price for the model you have selected. Keep model version, dimensions, normalization, and quantization settings alongside every index so you can compare and migrate without mixing incompatible vectors.

Shrinking vectors without losing meaning

At scale, storing and searching raw float32 vectors may exceed a practical memory or latency budget. Every exact scan must read $N \times d$ values; an approximate index reduces candidates, while quantization reduces bytes per stored candidate. Both are engineering choices that can lower recall, so each needs an evaluation gate.

Compression is a release experiment, not a one-way conversion. Start from a labeled FP32 baseline, reject any shortcut that loses required evidence from the shortlist, and keep authorization checks in the precise serving path.

Scalar quantization (SQ)

Scalar quantization (SQ) replaces each float32 (4 bytes) with a lower-precision representation, typically mapping each dimension independently. The simplest signed int8 version is symmetric around zero:

$$x_q = \operatorname{clip}\left(\operatorname{round}\left(\frac{x}{s}\right), -127, 127\right), \qquad x \approx s x_q$$

Here, s is the scale. Each dimension goes from 4 bytes (float32) to 1 byte (int8), so raw storage drops by about 4x while preserving an approximate reconstruction path back to float space. In practice, some libraries use this symmetric signed form, while others use affine quantization with a zero-point, often over uint8.

This function implements symmetric scalar quantization. It takes an array of float32 vectors and returns the compressed int8 vectors along with the per-dimension scales needed to approximately reconstruct the original floats later:

Quality impact

INT8 cuts the raw vector payload from four bytes per dimension to one. It doesn't promise any fixed Recall@K. Model distribution, calibration sample, scoring rule, shortlist depth, and reranking all change the observed loss, so treat INT8 as the first candidate to benchmark rather than an automatic release decision.

The outlier trap

The biggest practical mistake in scalar quantization is deriving the scale from the absolute minimum and maximum of a calibration sample without checking outliers. If one dimension spikes far outside the usual range, the scale stretches and ordinary values collapse into fewer integer buckets. Percentile clipping can preserve ordinary resolution while saturating outliers, but the percentile is another measured choice rather than a universal constant.

The next executable example quantizes ordinary values plus one spike. It compares a scale set by the spike with a clipped scale set from the ordinary calibration range:

Lower reconstruction error on ordinary values is encouraging, but retrieval is the real target. A release check must measure whether relevant approved passages stay in the shortlist after calibration and compression.

Product quantization (PQ)

Introduced by Jégou et al.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11}, Product Quantization (PQ) splits each vector into $M$ subvectors and quantizes each independently using a learned codebook.

Example with $M=4$ on an 8-dimensional vector:

$$\text{Original}: [1.2, 3.4 \mid 0.8, 2.1 \mid 5.3, 1.7 \mid 0.4, 3.2] \quad (8 \times 4 = 32 \text{ bytes})$$

This example shows a single 8-dimensional vector split into 4 subvectors of 2 dimensions each. Each subvector gets mapped to the ID of its nearest centroid in that subspace's codebook:

1. $[1.2, 3.4] \to \text{Centroid ID: } 7$
2. $[0.8, 2.1] \to \text{Centroid ID: } 23$
3. $[5.3, 1.7] \to \text{Centroid ID: } 3$
4. $[0.4, 3.2] \to \text{Centroid ID: } 41$

$$\text{Compressed code}: \quad (7, 23, 3, 41) \quad (4 \text{ bytes at 8 bits/code})$$

Each subvector is now stored as a centroid index instead of raw floats. Storage drops from 32 bytes to 4 bytes because we keep IDs; at query time, lookup tables score those IDs against the full-precision query.

With 256 centroids per subspace (8 bits), the stored code can be much smaller than the original vector. For example, compressing 768d float32 vectors (3,072 bytes) using PQ with 96 subspaces produces a 96-byte per-vector code, a 32x payload compression. Shared codebooks and index metadata add separate overhead.

Asymmetric distance computation (ADC)

At query time, you usually keep the query vector in full precision and compare each query subvector against the centroids in the matching codebook^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11}. That gives a small lookup table per subspace:

$$d(\mathbf{q}, \hat{\mathbf{x}}) \approx \sum_{m=1}^{M} \left\| \mathbf{q}_m - \mathbf{c}_{m, i_m} \right\|_2^2$$

The compressed database vector stores only centroid IDs $i_m$, so scoring a candidate becomes $M$ table lookups and additions. That's why PQ can search compressed vectors without fully decompressing them.

Inner-product and MIPS ADC

The squared-L2 form above is the classic PQ/ADC setup from Jégou et al.: you minimize summed subspace distances. Many embedding indexes serve maximum inner product search (MIPS) or unit-vector cosine (rank by $\langle q, x \rangle$). For those contracts, approximate scoring should maximize a sum of subspace inner products, not minimize L2:

$$\langle \mathbf{q}, \hat{\mathbf{x}} \rangle \approx \sum_{m=1}^{M} \langle \mathbf{q}_m, \mathbf{c}_{m, i_m} \rangle$$

Build a lookup table of $\langle \mathbf{q}_m, \mathbf{c}_{m,k} \rangle$ for each subspace $m$ and centroid $k$, then for each stored code sum the selected entries and rank high-to-low. When every vector is unit-normalized, this IP ADC approximates cosine ranking under compression. Do not train or evaluate L2 PQ tables against an unnormalized IP index and expect rankings to match: minimizing $\|q - \hat{x}\|^2$ is a different order from maximizing $\langle q, x \rangle$ unless norms are controlled. Production upgrades such as OPQ (rotate space before PQ so subspaces carry more independent signal) and residual PQ (quantize the residual after a coarse quantizer) improve recall without changing this "match the score you serve" rule.

In a real index, training data learns each codebook. Here we write down two tiny codebooks so you can see asymmetric distance computation (ADC) itself: the full-precision query creates a lookup table, and stored IDs select four distances to add.

Trade-off

Higher compression usually means lower recall. The exact loss depends on the number of subspaces, centroid count, and reranking budget, so benchmark PQ with your real query distribution instead of relying on a single headline number.

Binary quantization (BQ)

Of the three payload reductions here, Binary Quantization (BQ) is the most aggressive: it reduces each dimension to a single bit. This version thresholds at zero: positive numbers become 1, and zero or negative numbers become 0. It discards magnitude and granular precision, so a binary shortlist is a candidate to evaluate, not a final ranking guarantee.

The code snippet below keeps the transformation small. It takes one raw float32 vector and returns packed bytes containing one sign bit per dimension:

(vector > 0).astype(np.uint8) is still one byte per dimension. You only get true 1-bit storage once you pack those bits.

Worked example

Take a 4-dimensional embedding after L2 normalization: [0.3, -0.1, 0.8, -0.4]. Binary quantization thresholds at zero:

• Positive maps to 1, non-positive maps to 0, giving bit vector [1, 0, 1, 0]
• Packed into bytes with little-endian bit order: one byte holding those four bits, decimal value 5 (0b00000101).

Now compare against another binary vector [1, 0, 1, 1] (from [0.5, -0.2, 0.7, 0.1]):

• XOR the packed bytes, leaving differing bits only in the last position
• popcount (number of 1s in XOR result) = 1, so Hamming distance = 1

A Hamming distance of 1 out of 4 bits means the vectors are close in this binary approximation. In production, you can use this fast filter to pull a shortlist, then rescore survivors with INT8 or FP32 scores. Because sign bits discard magnitude, binary quantization always needs validation on your model and query distribution. This zero-threshold version works best when coordinates are centered around zero; other distributions need a different calibration or encoding choice.

In this packed-bit design, candidate scoring starts with Hamming distance (XOR + popcount), which is fast on modern CPUs because it relies on bitwise operations. It's only an approximation to angular similarity, though, so production systems often rescore the shortlist with a higher-precision representation.

The two-stage retrieval pattern

Production systems often combine quantization with reranking to balance speed and accuracy.

Two-stage retrieval works like triaging an incident-note backlog. Stage 1 (binary quantization) scans 10,000 notes for rough matches, which is fast but approximate. Stage 2 (full-precision reranking) gives the top 50 notes a detailed comparison. You wouldn't inspect all 10,000 in full because that's too slow, and you wouldn't resolve cases from rough matches alone because that's too inaccurate. The funnel gives you both speed and quality.

One cost-sensitive pattern stretches this into three stages: a binary Hamming search for the cheap shortlist, an INT8 dot-product rescore of that shortlist, and an optional cross-encoder reranker^{[4]Reference 4Rerank API.https://docs.cohere.com/reference/rerank} on the final handful when answer quality justifies the extra latency. Each stage looks at fewer candidates with more precision than the one before it.

The evidence-admission boundary from the previous lesson still applies after compression. This miniature release gate checks that binary retrieval keeps the approved policy in its shortlist, then applies precise scoring only to approved candidates:

On real data, run that same gate over many labeled queries and failure slices. A compressed first pass is acceptable only if its shortlist recall remains high enough for precise, authorized evidence selection to succeed.

Why is binary quantization usually paired with reranking instead of used as the final answer?

Answer

Binary quantization keeps only one sign bit per dimension, so Hamming distance is fast but coarse. It's useful for finding a shortlist cheaply. The final ranking should use a higher-precision representation, such as INT8 or FP32 dot product, and often a cross-encoder reranker when result quality matters.

Building a production pipeline

One production vector-search pattern stacks quantization for smaller payloads, an ANN index for lower search work, and reranking for a more precise final comparison. Exact composition is library-specific: some implementations combine HNSW with scalar or product quantization, while others use a different ANN structure or retain full-precision vectors alongside compressed codes for rescoring.

ANN indexing with HNSW

Many vector databases offer HNSW (Hierarchical Navigable Small World) indices^{[5]Reference 5Efficient and Robust Approximate Nearest Neighbor Using Hierarchical Navigable Small World Graphs.https://arxiv.org/abs/1603.09320} as an ANN structure. HNSW builds a multi-layered proximity graph: upper layers contain progressively fewer sampled nodes, while the base layer contains the indexed collection. Search starts in an upper layer, follows promising graph links, and descends toward a candidate neighborhood at the base.

When configuring an HNSW implementation, three controls commonly expose its quality/cost tradeoff. Exact names and allowed values depend on the library:

Measure recall and latency first while changing query-time exploration. If the graph can't reach the required recall within the latency budget, rebuild candidates with adjusted construction settings. No copied parameter triple proves quality for a new corpus.

Full pipeline overview

Model and dimension selection

Public benchmarks such as MTEB evaluate embedding models across several task families.^{[6]Reference 6MTEB: Massive Text Embedding Benchmark.https://arxiv.org/abs/2210.07316} They help you assemble candidates, but they can't tell you which index retrieves your approved policy passages under your storage and latency limits.

Keep a versioned experiment row for model, width, normalization, quantizer, index settings, shortlist size, and reranker. That row is what makes a compressed-index rollout explainable when an access-policy answer later changes.

Cost analysis at scale

As your vector corpus grows from millions to billions of documents, the cost structure of your retrieval system shifts sharply. At small scale, storing raw float32 vectors in RAM is feasible. At billion scale, it can become the dominant infrastructure cost because low-latency approximate nearest neighbor search keeps hot index data in memory.

The byte math is deterministic even when cloud pricing isn't. This table shows just the vector payload, rounded in binary units, before HNSW links, metadata, replicas, or region-specific RAM pricing:

Practice: choose a retrieval stack

You have 50 million approved policy and incident-resolution embeddings at 768 dimensions. A private draft-note collection must never be served as answer evidence. P99 latency budget is 60 ms, and raw FP32 vectors would crowd out the rest of the service's RAM.

1. Would you keep raw FP32 payload in memory for first-stage search?
2. If your ingest path already L2-normalizes vectors, would you score with cosine or dot product online?
3. Would you start with INT8 SQ, PQ, or BQ for the first pass?
4. Where would you spend full-precision compute?
5. What test prevents a cheaper index from silently changing evidence quality?

Solution sketch:

• Don't start with raw FP32 if memory is already tight. Benchmark INT8 first, then PQ if you still need a bigger cut.
• Use dot product online if vectors are already unit-normalized. It preserves cosine ranking without redundant norm work.
• Start by benchmarking INT8 SQ because it changes payload less aggressively than PQ or BQ. Move further only when the evaluated RAM/latency gain justifies measured shortlist loss.
• Spend full-precision compute on a shortlist, not on the whole corpus. That's where rescoring and reranking pay off.
• Build a labeled release set containing approved policies plus tempting unauthorized near-matches. Compare Recall@K against the full-precision baseline and reject any serving result sourced from unapproved text.

Mastery check

You have two unit-length vectors. Vector A has dot product 0.73 with Vector B. What is their cosine similarity, and why does this simplify production retrieval?

Answer

The cosine similarity is 0.73. Because both vectors have length 1, cosine similarity equals the dot product. Many production systems normalize once at ingest time, then use dot product for the rest of the pipeline. That avoids recomputing norms at query time while keeping the same ranking.

Key concepts

• Cosine similarity measures direction. Raw dot product measures direction plus magnitude.
• Once vectors are unit-normalized, cosine and dot product give the same ranking.
• Matryoshka Representation Learning trains selected prefix widths to remain useful; each deployed width still needs evaluation.
• Scalar quantization compresses each dimension independently. Product quantization compresses subvectors into centroid IDs.
• Binary quantization is a cheap filter, not a trustworthy final ranker.
• A safe retrieval rollout is staged and measured: compressed first pass, authorized shortlist check, higher-precision rescore, then optional reranker.

Evaluation rubric

• Strong: derives why normalized cosine and dot product are equivalent, then ties that to serving efficiency.
• Strong: computes rough RAM savings for FP32, INT8, and PQ without confusing bytes, GB, GiB, and code sizes.
• Strong: explains PQ as subvector codebooks plus ADC lookup tables, not as generic compression.
• Strong: chooses compression stage based on latency, evidence-recall target, and RAM budget, then defines a release gate.
• Weak: says "always use cosine" or "higher dimensions are always better" without checking normalization or workload.
• Weak: treats quantization as inherently safe or inherently inaccurate instead of measuring shortlist recall and served evidence.

Follow-up questions

Why does dot product become equivalent to cosine similarity for normalized vectors?

Cosine divides the dot product by the product of the vector norms. If both vectors have norm 1, that denominator is 1, so cosine similarity equals the dot product. That lets a retrieval system normalize once, then use fast inner-product scoring without recomputing norms.

How does Matryoshka Representation Learning differ from PCA?

MRL is a training-time technique that teaches one embedding to work at several prefix lengths. PCA is a post-training projection applied to frozen vectors. MRL is useful when the model was trained for prefix truncation; PCA is a retrofit when you already have embeddings and need a smaller projected representation.

Why is Product Quantization often more attractive than Scalar Quantization at billion-vector scale?

Scalar quantization usually gives about 4x raw payload compression by moving from float32 to int8. Product Quantization can compress much more aggressively by storing centroid IDs for subspaces, which can be the difference between a huge distributed RAM footprint and a smaller compressed index. PQ usually needs rescoring because it introduces approximation error.

Why is Binary Quantization faster than float32 dot products?

Binary Quantization stores one sign bit per dimension, packs bits into bytes, and scores candidates with XOR plus popcount. That cuts memory traffic and replaces floating-point multiply-adds with cheap bit operations, but the score is coarse and should usually feed a reranker.

When is bigger embedding dimension worth the extra RAM bill?

Only when your own query set shows that smaller prefixes or lower-dimensional models miss too many relevant items even after reranking. Dimension count is a measured tradeoff, not a prestige number.

Common pitfalls

• Recall falls after INT8 rollout. Cause: scale got stretched by outliers or mismatch between offline and online normalization. Fix: clip calibration range robustly, recheck normalization contract, and benchmark recall on real queries.
• ANN results look fine offline but wrong in production. Cause: you tuned on MTEB or synthetic samples instead of real ticket language and chunking. Fix: build an eval set from your own queries and review misses manually.
• Search is still too slow after quantization. Cause: you compressed payload but still rescore too many candidates or set ef_search too high. Fix: tune shortlist size, HNSW parameters, and rerank depth together.
• Migration silently degrades quality. Cause: old and new vectors use different model versions, dimensions, or quantization settings. Fix: version the whole embedding contract and build the new index side by side before traffic cutover.