Embedding Similarity & Quantization

An operator asks, How do I rotate an API key? The embedding model should put that query near approved api-key-rotation-v3 and away from a quota page. The system still has to put the guide first, fit millions of vectors in memory, and never turn private-incident-note-44 into answer evidence.

That gives us one evidence boundary to protect. The private note can mention API keys and still be unauthorized. A raw dot-product search can reward its larger norm even when its direction is worse.

A separate capacity shortcut can drop the approved guide from a compressed shortlist. Neither mistake is obvious if you inspect only the final answer, so we'll keep both failures visible as we move through the index.

Sentence embeddings showed how contrastive training shapes this space. Here the encoder stays fixed. We'll keep both chunks in view while we choose a scoring contract, a width, and a representation that preserve the right evidence.

Follow one question: can api-key-rotation-v3 survive every cheaper shortcut? We'll compute three notions of closeness, test normalization and shorter prefixes, size float and compressed vectors with scalar, product, and binary quantization. Then we'll add approximate search and a final approval gate.

Three ways to measure closeness

Start with one small question: what should "close" mean for these two chunks? An embedding is a point in high-dimensional space, represented by a list of numbers from the encoder. Three common rulers answer different versions of that question.

The table becomes easier to use once you make the contract explicit. If every stored vector and query is normalized to length 1, all three rulers rank candidates in the same order. You can then choose the scoring kernel your index runs quickly, but you must apply the same normalization at ingest and query time.

A zero vector is the exception you have to reject explicitly: cosine divides by both vector norms, so a zero-norm query has no defined direction. Replacing that division with an arbitrary zero score can quietly reorder an entire shortlist. Record the encoder failure and refuse retrieval instead of pretending the query matched nothing.

A tiny 2D example makes the split visible. Two paraphrases of the rotation guide map to:

• Passage A, api-key-rotation-v3: [3, 4]
• Passage B, a second rotation phrasing: [4, 3]

Both point northeast, so predict that every ruler will call them close. Work the arithmetic by hand before touching code.

Keep this equal-length pair in mind. Later fixtures will make the less obvious case visible: same direction, different length.

Cosine similarity

For A and B, the shared direction should produce a high score even though the score must ignore their lengths. Cosine keeps only that direction:

$$\cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}$$

Start with the numerator. Multiply matching dimensions and add:

$$\mathbf{A} \cdot \mathbf{B} = (3 \times 4) + (4 \times 3) = 12 + 12 = 24$$

The length of a vector is the square root of the sum of squared components:

$$\|\mathbf{A}\| = \sqrt{3^2 + 4^2} = \sqrt{25} = 5$$ $$\|\mathbf{B}\| = \sqrt{4^2 + 3^2} = \sqrt{25} = 5$$

Their lengths are both 5, so now divide by $5 \times 5$:

$$\cos(\theta) = \frac{24}{5 \times 5} = \frac{24}{25} = 0.96$$

1 means the same direction, 0 means orthogonal, and -1 means opposite. That's a geometry result, not a relevance guarantee. Whether the direction tracks useful matches still depends on the encoder and retrieval evaluation.

Dot product

Now remove the length correction. Dot product stops at the sum we already computed:

$$\mathbf{A} \cdot \mathbf{B} = \sum_{i=1}^{n} a_i b_i = 24$$

These passages happen to have the same length, so dot and cosine agree. If A gets longer without changing direction, dot product rises while the angle stays put. That's useful when training made magnitude meaningful. It's a bug when your contract was angular.

Euclidean distance

The third ruler asks for the straight-line gap. Subtract, square, sum, and take the square root:

$$\|\mathbf{A} - \mathbf{B}\|_2 = \sqrt{(3-4)^2 + (4-3)^2} = \sqrt{2} \approx 1.41$$

A small L2 distance also says these paraphrases are close. For this pair, all three rulers agree. Add candidates with different norms and that agreement can disappear, which is why the serving contract matters more than the label on an index.

You've worked the numbers. Predict the next output: after A is scaled by 10, should cosine change, should dot product change, or should both stay fixed? This snippet checks the hand math and makes that prediction concrete:

If vector A is [3, 4] and vector C is [30, 40], what happens to cosine similarity and dot product when you compare both vectors to [4, 3]?

Answer

Cosine similarity stays the same because A and C point in the same direction. Dot product grows by 10x because C is 10x longer. Cosine ignores magnitude. Dot product keeps it.

The experiment gives you a serving decision. Use cosine when the model card, training setup, or labeled queries say length shouldn't change rank. Use raw dot product when magnitude is part of the learned score, as in maximum inner product search (MIPS).

If every vector is already unit length, the rankings match and inner product avoids norm divisions while scoring candidates.

When the three rulers agree

The first fixture used equal-length vectors. Real encoders don't promise that unless their documentation or your pipeline says so. Treat normalization as an explicit contract shared by model output, index construction, and query serving. Don't infer it from a model name.

With that contract in place, the three rulers collapse to one ranking. When both vectors have length 1, the cosine denominator is $1 \times 1 = 1$, so cosine equals raw dot product. Squared Euclidean distance becomes a strictly decreasing function of that same inner product:

$$\cos(\mathbf{A}, \mathbf{B}) = \mathbf{A} \cdot \mathbf{B} = 1 - \frac{\|\mathbf{A} - \mathbf{B}\|^2}{2}$$

Higher inner product, higher cosine, and smaller squared L2 distance now induce the same ranking. Normalize stored rows once and each incoming query once. Candidate scoring can use a matrix-multiply kernel without repeated norm divisions.

Here's the failure in the opening story. The next fixture gives private-incident-note-44 a huge raw norm, so it wins unnormalized dot product even though its direction is less aligned with the query. Before running it, predict which label each score will choose. Then compare raw dot, normalized dot, and cosine.

A large-norm vector isn't automatically junk. The narrower conclusion is about serving: if the contract is cosine-like ranking, skipping normalization changes what the index optimizes.

L2 distance isn't automatically wrong in high dimension either. On unit vectors, squared L2 produces the same ranking through the identity above. Without normalization, L2 also includes norm differences. Keep it when the model, index, and evaluation were built around that geometry.

The original product-quantization formulation approximates this squared L2 distance between a full-precision query and compressed database vectors.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11}

We've fixed what "close" means. The next pressure is capacity: how many coordinates can you afford to keep before the scoring contract becomes too expensive?

Choosing how many dimensions you need

We've chosen a scoring contract. Now ask what that contract costs. Width controls how much of the trained representation you store, along with memory traffic and scoring work.

A dimension count has no universal quality meaning: a 512-dimensional model trained for your API-doc queries can beat a different 1,536-dimensional model. Compare widths only after holding model family, scoring convention, corpus, and evaluation set fixed.

The table separates two decisions that are easy to conflate. Dimension count changes capacity. Normalization changes score meaning. A supported shorter vector still needs the serving normalization contract documented by its model or applied by your pipeline.

For one representation family, extra dimensions buy capacity but charge the same predictable systems costs:

1. Storage. Each float32 is 4 bytes. 1M vectors at 1536d is about 5.7 GiB of raw payload.
2. Exact scoring work. A dot product or L2 comparison does linear work in the number of dimensions, written $O(d)$, per candidate.
3. Memory bandwidth. Loading vectors from memory can bottleneck approximate nearest neighbor (ANN) search, which finds close vectors without scanning every row.
4. End-to-end latency. Halving dimensions doesn't promise halved latency. Graph traversal, filters, networking, and reranking still take time, so measure before you pay or save the memory bill.

Matryoshka prefixes still need a recall test

Suppose memory pressure makes you want 256 dimensions instead of 1,536. Should the first 256 coordinates be useful by accident, or should the model have been trained to put signal there?

Matryoshka Representation Learning (MRL) answers the second question: training minimizes a weighted sum of losses at selected truncation widths $d \in \mathcal{D}$, so those prefixes are meant to work on their own.^{[2]Reference 2Matryoshka Representation Learning.https://arxiv.org/abs/2205.13147}

That's different from post-hoc PCA, which projects already-trained vectors into a smaller linear subspace, usually by preserving variance on a fixed dataset. PCA can still help frozen embeddings. MRL fits a serving path with several dimension budgets when the model was trained for truncation.

Hosted APIs can expose a supported shortening control without you slicing tensors by hand. OpenAI's embeddings guide documents a dimensions parameter for text-embedding-3 models, with defaults of 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large.^{[3]Reference 3Vector embeddingshttps://developers.openai.com/api/docs/guides/embeddings}

The same guide reports that a text-embedding-3-large vector shortened to 256 dimensions outperforms unshortened text-embedding-ada-002 at 1,536 dimensions on MTEB. That's a provider benchmark, not a promise that a 256-d prefix will retrieve api-key-rotation-v3 for your queries.

The provider's shortening control and a local slice aren't interchangeable. If you cut a prefix from an already generated vector, normalize that sliced prefix before indexing, then apply the same operation to queries. OpenAI's embeddings guide shows this L2 renormalization after a manual cut.^{[3]Reference 3Vector embeddingshttps://developers.openai.com/api/docs/guides/embeddings}

This local check enforces the contract without an API call:

Normalization restores a unit-vector scoring contract. It can't restore coordinates you threw away. In the next fixture, the last two dimensions carry the distinction between the two passages. Cut to width 2 and the private note ties the guide, so argmax picks the unauthorized row. Predict that result before reading the output.

This isn't a benchmark for a real MRL model. It's the shape of a release test: evaluate each supported width on approved-query fixtures, including near-duplicate unauthorized passages. A prefix that saves bytes but loses the guide fails the contract.

You shorten a unit-normalized 3072-dimensional embedding by taking only the first 256 values. Should you index that prefix immediately?

Answer

No. Slicing usually changes the vector's L2 norm, so normalize the 256-dimensional prefix before indexing. Otherwise cosine, dot product, and Euclidean conventions no longer match the index contract you think you are using.

The memory problem: a sizing example

The prefix test exposed a quality budget. Now put a number on the memory budget. Use the default text-embedding-3-small width of 1,536 dimensions and ask what one million rows cost before graph links, metadata, or replicas.

Scenario: 1 million embeddings, 1,536 dimensions, stored as 32-bit floats (FP32). Predict the raw payload first. Then we'll replace four-byte floats with one-byte values and finally with PQ IDs.

Step 1: the FP32 baseline

Each dimension is a float32, which takes 4 bytes. Multiply rows, dimensions, and bytes per value:

$$1{,}000{,}000 \times 1{,}536 \times 4 \text{ bytes} = 6.144 \times 10^9 \text{ bytes} \approx 6.14 \text{ GB} \approx 5.72 \text{ GiB}$$

That's only the raw numbers. Real indexes add graph links, metadata, and replicas, but the vector payload alone is already over 5.7 GiB. If that's too large for hot memory, precision is the next lever.

Step 2: 8-bit scalar quantization

Map each float32 to a single byte (INT8). The row count and width stay fixed, so storage drops by about 4x:

$$1{,}000{,}000 \times 1{,}536 \times 1 \text{ byte} \approx 1.54 \text{ GB} \approx 1.43 \text{ GiB}$$

Step 3: product quantization

Split each 1,536-dimensional vector into 96 subvectors and store one 8-bit centroid ID per subvector. The compressed row is 96 bytes:

$$1{,}000{,}000 \times 96 \text{ bytes} \approx 91.6 \text{ MiB}$$

That's a 64x cut from the original FP32 payload. The trade is now explicit: you're comparing approximate codes instead of exact floats, so pair PQ with a higher-precision reranking stage.

The byte math is easy to turn into a sizing helper. Predict the units before running it: decimal gigabytes divide by $10^9$, while operating systems usually report binary gibibytes divided by $1024^3$:

Re-embedding cost scales with total input tokens, not document count alone. Compute it as total_tokens / 1_000_000 * embedding_price_per_1M_tokens, using the current price for the model you selected.

Keep model version, dimensions, normalization, and quantization settings alongside every index. Otherwise a migration can mix vectors that no longer share a scoring contract.

Shrinking vectors without losing the hit

The sizing math says where the pressure comes from. An exact scan touches up to $N \times d$ values. An ANN index reduces how many rows you visit, while quantization reduces bytes per row. Those are separate shortcuts, and either can lose the approved guide, so each needs its own evaluation gate.

Read the flow as a release decision, not a compression recipe. If the FP32 baseline fits RAM and latency, keep it. If it doesn't, benchmark one shortcut at a time, reject any version that drops required evidence from the shortlist, and keep authorization on the precise serving path.

Scalar quantization (SQ)

Start with the least structural change. Scalar quantization replaces each float32 (4 bytes) with a lower-precision number, typically mapping each dimension independently. The simplest signed int8 version is symmetric around zero:

$$x_q = \operatorname{clip}\left(\operatorname{round}\left(\frac{x}{s}\right), -127, 127\right), \qquad x \approx s x_q$$

Here $s$ is the scale. Each dimension goes from 4 bytes to 1 byte, so raw storage drops by about 4x while you keep an approximate path back to float space. Some libraries use this symmetric signed form; others use affine quantization with a zero-point, often over uint8. The code below makes the scale and reconstruction error inspectable.

This function implements the symmetric version. It takes float vectors and returns integer codes plus the per-dimension scales needed to reconstruct later:

INT8 cuts the raw payload from four bytes per dimension to one. It doesn't promise any fixed Recall@K. Model distribution, calibration sample, scoring rule, shortlist depth, and reranking all change the observed loss.

Treat INT8 as the first candidate to benchmark, not an automatic release.

⚠️ Common mistake: Deriving the scale from the absolute min and max of a calibration sample without checking outliers. If one dimension spikes far outside the usual range, the scale stretches and ordinary values collapse into fewer integer buckets. Percentile clipping can preserve ordinary resolution while saturating outliers, but the percentile is another measured choice, not a universal constant.

The next example isolates that failure with ordinary values plus one spike. Predict which calibration range gives ordinary values more resolution, then compare a scale set by the spike with a clipped scale set from the ordinary range:

Lower reconstruction error on ordinary values is encouraging, but retrieval is the target. A release check must measure whether relevant approved passages stay in the shortlist after calibration and compression.

Product quantization (PQ)

SQ keeps one approximate number for every coordinate. If that still leaves too many bytes, PQ changes the unit of storage: instead of storing coordinates, store the IDs of nearby codewords. Jégou, Douze, and Schmid introduced this pattern by splitting each vector into $M$ subvectors and quantizing each independently with a learned codebook.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11}

See the storage change in a small example with $M=4$ on an 8-dimensional vector:

$$\text{Original}: [1.2, 3.4 \mid 0.8, 2.1 \mid 5.3, 1.7 \mid 0.4, 3.2] \quad (8 \times 4 = 32 \text{ bytes})$$

Each 2-d chunk maps to the nearest centroid ID in that subspace's codebook. Stored as 8-bit IDs, the row becomes four bytes. At query time you don't reconstruct the floats. You score those IDs against the full-precision query with lookup tables.

With 256 centroids per subspace (8 bits), compressing 768-d float32 vectors (3,072 bytes) using 96 subspaces produces a 96-byte code, a 32x payload cut. Shared codebooks and index metadata add separate overhead.

Asymmetric distance computation (ADC)

The asymmetry is deliberate: keep the query in full precision, and compare each query subvector with every centroid in its matching codebook.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11} That builds a small lookup table per subspace:

$$d(\mathbf{q}, \hat{\mathbf{x}}) \approx \sum_{m=1}^{M} \left\| \mathbf{q}_m - \mathbf{c}_{m, i_m} \right\|_2^2$$

The database row stores only centroid IDs $i_m$. Scoring one candidate becomes $M$ table lookups and additions, so PQ can search compressed vectors without fully decompressing them. Production libraries such as FAISS implement this ADC path for PQ codes.^{[4]Reference 4Billion-scale similarity search with GPUs.https://arxiv.org/abs/1702.08734}

The distance equation above minimizes summed subspace distances, so it matches an L2 serving contract. Many embedding indexes serve maximum inner product search (MIPS) or unit-vector cosine, where higher $\langle q, x \rangle$ wins. For those contracts, use a sum of subspace inner products instead:

$$\langle \mathbf{q}, \hat{\mathbf{x}} \rangle \approx \sum_{m=1}^{M} \langle \mathbf{q}_m, \mathbf{c}_{m, i_m} \rangle$$

Build a lookup table of $\langle \mathbf{q}_m, \mathbf{c}_{m,k} \rangle$ for each subspace $m$ and centroid $k$. For each stored code, sum the selected entries and rank high-to-low. When every vector is unit-normalized, IP ADC approximates cosine ranking under compression.

Don't train or evaluate L2 PQ tables against an unnormalized IP index and expect the same order: minimizing $\|q - \hat{x}\|^2$ differs from maximizing $\langle q, x \rangle$ unless norms are controlled.

OPQ rotates the space before PQ so subspaces carry more independent signal.^{[5]Reference 5Optimized Product Quantizationhttps://openaccess.thecvf.com/content_cvpr_2013/html/Ge_Optimized_Product_Quantization_2013_CVPR_paper.html} Residual / IVFADC pipelines quantize a residual after a coarse quantizer.^{[1]Reference 1Product Quantization for Nearest Neighbor Search.https://dblp.org/rec/journals/pami/JegouDS11} Either can raise recall. Neither changes the rule: match the score you serve.

The next snippet makes ADC concrete with two-centroid codebooks. The full-precision query creates four lookup tables; each stored ID selects one entry per table. Predict which code should win before adding those four distances.

More compression can lower recall. The loss depends on subspace count, centroid count, query distribution, and reranking budget. Benchmark PQ against the real query distribution instead of trusting one headline number.

Binary quantization (BQ)

If one byte per coordinate still misses the budget, binary quantization keeps only one bit per dimension. This version thresholds at zero: positive numbers become 1, and zero or negative numbers become 0. Magnitude disappears. That makes BQ attractive for a very cheap shortlist, not a final ranking guarantee.

Pack the sign bits into an integer. Comparing two packed rows then becomes XOR plus popcount, which gives Hamming distance:

Predict the count before reading the output: the two vectors differ only at the last sign. [0.3, -0.1, 0.8, -0.4] becomes bits [1, 0, 1, 0]. Little-endian packing puts dimension 0 in bit 0, so the integer is 0b0101 = 5. The second vector becomes [1, 0, 1, 1] = 13. XOR leaves one differing bit, so Hamming distance is 1.

A Hamming distance of 1 out of 4 bits means the vectors are close in this binary approximation. Sign bits discard magnitude, so this zero-threshold version is most informative when coordinates are centered around zero. Other distributions need different calibration or encoding, and the first stage still needs a recall test.

Shortlist cheaply, then rescore

The three representations now have distinct jobs. Stage 1 scans a large candidate set with a cheap code. Stage 2 gives a shortlist a precise comparison. You wouldn't inspect 10,000 API-doc chunks in full precision on every query, and you wouldn't cite the top Hamming match as evidence.

One cost-sensitive pattern uses three stages: binary Hamming search for the cheap shortlist, an INT8 dot-product rescore of that shortlist, and an optional cross-encoder reranker on the final handful when answer quality justifies the extra latency.^{[6]Reference 6Rerank API.https://docs.cohere.com/reference/rerank}

Each stage sees fewer candidates and more precision than the one before it.

Compression doesn't move the evidence boundary. Hamming can tie a private note with the rotation guide. Unit cosine can separate them, but the approval filter still has to drop the note.

On real data, run that same gate over many labeled queries and failure slices. A compressed first pass earns its place only if shortlist recall stays high enough for precise, authorized evidence selection to succeed.

Why is binary quantization usually paired with reranking instead of used as the final answer?

Answer

Binary quantization keeps only one sign bit per dimension, so Hamming distance is fast but coarse. It's useful for finding a shortlist cheaply. The final ranking should use a higher-precision representation, such as INT8 or FP32 dot product, and often a cross-encoder reranker when result quality matters.

Indexing the compressed codes

Compression answers "how many bytes per row?" ANN answers "how many rows do we visit?" Keep those levers separate. Exact composition is library-specific: some implementations combine HNSW with scalar or product quantization, while others keep full-precision vectors beside compressed codes for rescoring.

Many vector databases offer HNSW (Hierarchical Navigable Small World) graphs as an ANN structure.^{[7]Reference 7Efficient and Robust Approximate Nearest Neighbor Using Hierarchical Navigable Small World Graphs.https://arxiv.org/abs/1603.09320} HNSW builds a multi-layered proximity graph: upper layers contain progressively fewer sampled nodes, while the base layer contains the collection.

Search starts high, follows promising links, and descends toward a candidate neighborhood at the base.

If a compressed index still misses the guide, you need to know which knob can change without rebuilding. HNSW implementations commonly expose three controls; exact names and allowed values depend on the library:

Start with query-time exploration and measure recall and latency together. If the graph can't reach required recall within the latency budget, rebuild with adjusted construction settings. A copied parameter triple says nothing about a new corpus.

Public benchmarks such as MTEB evaluate embedding models across several task families.^{[8]Reference 8MTEB: Massive Text Embedding Benchmark.https://arxiv.org/abs/2210.07316} They help you assemble candidates. They can't tell you which index retrieves api-key-rotation-v3 under your storage, authorization, and latency limits.

Keep a versioned experiment row for model, width, normalization, quantizer, index settings, shortlist size, and reranker. That row lets you explain a compressed-index rollout when a rotation-guide answer later changes.

Bytes, RAM, and the shortlist bill

The one-million-row example hides scale. As the corpus grows from millions to billions of chunks, low-latency ANN search keeps more index data hot in memory, and raw float32 payload can dominate the bill. The ratios stay simple, so use the table to choose where to measure next.

The byte math is deterministic even when cloud pricing isn't. This table uses the same 1,536-d width as the sizing example, rounded in binary units, before HNSW links, metadata, replicas, or region-specific RAM pricing:

Practice: choose a retrieval stack

You have 50 million approved API-doc embeddings at 768 dimensions. private-incident-note-44 must never be served as answer evidence. P99 latency budget is 60 ms, and raw FP32 vectors would crowd out the rest of the service's RAM. Choose a path that saves memory without weakening the evidence boundary.

1. Would you keep raw FP32 payload in memory for first-stage search?
2. If your ingest path already L2-normalizes vectors, would you score with cosine or dot product online?
3. Would you start with INT8 SQ, PQ, or BQ for the first pass?
4. Where would you spend full-precision compute?
5. What test prevents a cheaper index from silently changing evidence quality?

Use this as a design review. Give each answer a reason tied to the scoring contract, memory budget, or evidence gate.

Solution sketch:

• Don't start with raw FP32 if memory is already tight. Benchmark INT8 first, then PQ if you still need a bigger cut.
• Use dot product online if vectors are already unit-normalized. It preserves cosine ranking without redundant norm work.
• Start by benchmarking INT8 SQ because it changes payload less aggressively than PQ or BQ. Move further only when the measured RAM and latency gain justifies shortlist loss.
• Spend full-precision compute on a shortlist, not on the whole corpus. That's where rescoring and reranking pay off.
• Build a labeled release set containing approved rotation and token-lifecycle docs plus tempting unauthorized near-matches. Compare Recall@K with the full-precision baseline, and reject any serving result sourced from unapproved text.

Common pitfalls

When a release goes wrong, the symptom usually points to one boundary. Use it to choose the next measurement instead of changing every knob at once.

• Recall falls after INT8 rollout. Cause: scale got stretched by outliers, or offline and online normalization disagree. Fix: clip the calibration range robustly, recheck the normalization contract, and benchmark recall on real queries.
• ANN results look fine offline but wrong in production. Cause: you tuned on MTEB or synthetic samples instead of real ticket language and chunking. Fix: build an eval set from your own queries and review misses manually.
• Search is still too slow after quantization. Cause: you compressed payload but still rescore too many candidates or set ef_search too high. Fix: tune shortlist size, HNSW parameters, and rerank depth together.
• Migration silently degrades quality. Cause: old and new vectors use different model versions, dimensions, or quantization settings. Fix: version the whole embedding contract and build the new index side by side before traffic cutover.