Vision-Language Models & CLIP

One inspection photo can raise two very different questions. A technician uploads a rack picture and types "blocked vent on rack door." The archive should find similar inspections, but the same technician might then ask for the printed asset tag or for a box around the hazard. A plausible match isn't enough evidence to close a safety ticket, so the system needs a cheap way to find candidates and a deliberate way to inspect them.

Contrastive Language-Image Pre-training (CLIP) gives us the cheap first pass: it places images and captions in one embedding space, so photos can be indexed offline and searched with text.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} The rest of this design follows one rack photo from that score to the evidence a reviewer can trust.

Product brief: data-center visual inspection

The service lets technicians upload rack photos, search past inspections in natural language, and route uncertain hazards to a reviewer. CLIP or Sigmoid Loss for Language-Image Pre-training (SigLIP) powers first-pass retrieval. Optical character recognition (OCR), grounding, or a generative vision-language model (VLM) handles questions that need label text, regions, or explanations. That split keeps a similarity score from pretending to be a complete inspection.

Requirements and API

• POST /v1/inspection-images accepts an immutable image reference, site, rack, capture time, and idempotency key. It returns image_id, ingestion state, and encoder version.
• POST /v1/inspection-search accepts query, requested site filters, and limit. The server intersects those filters with authenticated permissions. It returns ranked image_id values, similarity scores, thumbnails, and review state. Scores are ranking signals, not probabilities; clients can't lower a safety-review threshold.
• POST /v1/inspection-analysis accepts image_id, question, and requested evidence type. It returns answer, OCR spans or grounded regions, model versions, and answered, abstained, or review.
• Illustrative service targets are p95 below 300ms for search and below 3 seconds for analysis, not measured results. Access filters apply before retrieval, and a hazard never auto-closes from an unreviewed model result.

Data flow and sizing

Ingestion has one job: turn an authorized image into a searchable, versioned record. It follows upload -> malware and format checks -> metadata authorization -> image encoder -> normalized embedding -> versioned vector index. A failed check stops before the image becomes searchable.

Search has a different job. It follows authorized query -> text encoder -> site-filtered approximate nearest neighbor (ANN) search -> optional reranker -> response, with access filters applied before candidates are returned. Analysis starts only after the selected image is authorized, then follows crop or tile policy -> OCR or VLM -> grounding check -> review policy. Protect search capacity from bursts of expensive analysis. Separate GPU pools are one option; sharing needs admission limits and scheduling that meet the search target under mixed load.

Start with the vector bytes. For 10 million images and 768-dimensional 16-bit floating-point (FP16) embeddings, raw storage is 10,000,000 x 768 x 2 = 15.36 GB. That number excludes index structures, metadata, replicas, and old encoder versions.

The selected hierarchical navigable small-world (HNSW) graph or inverted-file (IVF) partitions determine the working set. An assumed total multiplier of 2 to 4 times raw storage would mean roughly 31 to 61 GB per generation, not a measured index requirement. Some indexes retain FP32 vectors despite FP16 inputs. Measure the actual vectors, graph, IDs, filters, replicas, and simultaneous rollout generations before sizing hosts.

At 100 queries per second (QPS), give the text encoder and ANN query path their own pool. Graph probes and encoder work shouldn't compete with the separately sized generative-analysis pool.

Recovery, rollout, and evaluation

Bind each idempotency key to the authenticated tenant and immutable request fingerprint; a content hash supports deduplication, not permission sharing. Reject a reused key with different image or metadata. Failed images enter retry or dead-letter state without becoming searchable. An encoder upgrade writes a new index generation; switch the paired text encoder, image preprocessing, normalization, and index generation together after validation. Equal vector dimensions don't make different models compatible. Retain the previous complete generation for rollback.

Treat uploads and image text as untrusted data. Bound decoded dimensions and tile counts as well as compressed bytes; resolve only authorized object references instead of fetching arbitrary user URLs. Strip unnecessary location metadata, restrict thumbnails and OCR artifacts, and apply deletion to indexes, crops, caches, and source images. Printed instructions such as "ignore safety policy" remain pixels or OCR evidence, not authority for tool calls. Recheck authorization before analysis and before returning cached results.

This in-memory example demonstrates the access boundary, not an ANN implementation. permitted comes from authenticated server state, not the request body. A requested site can't expand it, and a previously returned ID doesn't retain access after revocation.

def candidate_ids(records: dict, requested: set[str], permitted: set[str], generation: str) -> list[str]:
    visible_sites = requested & permitted
    return [image_id for image_id, row in records.items()
            if row["site"] in visible_sites and row["generation"] == generation]

def read_image(records: dict, image_id: str, permitted: set[str]) -> str:
    row = records.get(image_id)
    if row is None or row["site"] not in permitted:
        raise PermissionError("image unavailable")
    return row["object_version"]

records = {
    "a": {"site": "east", "generation": "clip-v1", "object_version": "a:v3"},
    "b": {"site": "west", "generation": "clip-v1", "object_version": "b:v1"},
    "c": {"site": "east", "generation": "clip-v2", "object_version": "c:v1"},
}
print("visible candidates:", candidate_ids(records, {"east", "west"}, {"east"}, "clip-v1"))
print("authorized object:", read_image(records, "a", {"east"}))
try:
    read_image(records, "a", set())  # permission was revoked after search
except PermissionError as error:
    print("after revocation:", error)

visible candidates: ['a']
authorized object: a:v3
after revocation: image unavailable

Analysis needs its own safe failure. If OCR or grounding fails, return abstained or review rather than a caption presented as verified evidence. A fluent sentence is still a model output, not a measurement.

Roll out with a frozen, technician-reviewed set, shadow traffic, a site-limited canary, and staged index switching. Block expansion when asset retrieval recall@10, hazard false-negative rate, asset-tag OCR accuracy, grounding IoU, p95 latency, review rate, or cross-site access tests miss their route-specific bars. Monitor drift by site, camera, lighting, rack type, and prompt template instead of relying on one aggregate benchmark.

Many supervised computer vision systems expose a fixed label inventory: recognizing a new category generally requires labeled examples and another training step. CLIP showed that image-text matching at web scale can produce representations that transfer to new text-defined classes without training a new classifier head.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} That made practical open-vocabulary retrieval possible and provided useful vision features for later generative systems.

That gives us a routing plan, not yet a trustworthy score. Follow the uploaded rack photo into the shared space first; then we can see which questions need a different model.

Score the rack photo with text first

Embeddings are the first inspection index

Suppose the search box contains "blocked vent on rack door." Each stored photo has already become a 768-number embedding, and the query becomes another vector of the same size. The index doesn't need to compare pixels with words at request time; it ranks stored image vectors against the query vector.

If the model has learned the right visual signal, matching image and text vectors point in roughly the same direction. Cosine similarity measures that alignment with a score between -1 and 1. Within one trained model, a higher score ranks a pair as more aligned than a lower-scoring candidate.

The number still isn't a calibrated probability, and it doesn't prove that the image contains every detail in the caption. CLIP's contract is narrower: rank matching image-text pairs above mismatches.

Why rack_hazard is a bad first index

A fixed label list can teach one class called rack_hazard, but that class says little about how blocked airflow looks across lighting, camera angles, cables, and rack labels. What happens when a technician adds a new phrase next week?

Learning from many image-text pairs gives the model a broader signal. It learns to align a visual concept with a natural-language description, rather than memorizing only the names in one classifier head.

The shared space lets the system score text prompts that weren't fixed training labels. Transfer still depends on whether pre-training learned the relevant evidence and whether the new domain resembles its data. Validate prompts on the team's racks, lighting, cabling, and review policy before turning a score into an operational decision.

Shared image-text space

The shared map is the handoff between two different inputs. One encoder starts with pixels and the other with words, but both are trained so a matched pair receives a higher similarity than a mismatch. The original model learned this from 400 million image-text pairs.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

That makes a prompt such as "blocked airflow behind rack panel" possible even when the phrase wasn't a training label. It remains a transfer hypothesis: test it on the racks that matter before trusting the rank.

Two encoders, one rack-photo score

Architecture[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020

What has to happen inside the model for that shared map to exist? CLIP jointly trains an image encoder and a text encoder, so a rack photo and "blocked vent on rack door" land in one coordinate system. The original paper used either a ResNet (Residual Network) or a ViT (Vision Transformer) for images and a Transformer for text, producing one representation for each caption.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

Before the figure, predict the matching point: do the two encoders exchange pixels and words layer by layer, or do they meet after encoding? They meet after encoding. A contrastive objective pulls matched representations together and pushes mismatches apart.

The two lanes stay separate until their outputs are normalized and compared in the shared map. That separation is what lets us encode the image once, store it, and answer many text queries later.

Implementation details remain checkpoint-specific. The official CLIP repository's encode_image and encode_text return unnormalized features. Normalize both before using their dot product as cosine similarity; the model's full forward does this internally and applies its learned logit scale. The README's separate zero-shot example uses a fixed scale of 100, so distinguish that example from the learned-scale implementation.^{[2]Reference 2CLIP: Official Implementationhttps://github.com/openai/CLIP}

Its tokenizer defaults to a 77-token sequence including start/end tokens. Overlong input raises by default; setting truncate=True cuts it and restores the end token, which can discard a decisive phrase. Changing context_length alone doesn't extend a checkpoint's learned positional embeddings. Use the matching preprocessing returned by clip.load, and bind it, both encoders, normalization, and the similarity metric to the index generation. These source details were checked October 10, 2026; no pretrained model was run here.^{[2]Reference 2CLIP: Official Implementationhttps://github.com/openai/CLIP}

Training objective

Training turns the shared map into a ranking game. For every image, the model must pick its paired caption from the other captions in the batch. The reverse direction asks whether each caption can find its paired image.

Use a tiny batch so the game stays visible. Suppose you have 3 images and 3 captions:

There are $3 \times 3 = 9$ possible pairings. Only the 3 diagonal pairs (A+A, B+B, C+C) are correct. The other 6 are mismatches. Before reading the labels in the figure, predict what training should do: raise the diagonal scores and lower the six off-diagonal scores.

The real batch in the original paper contains 32,768 pairs.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} Each image therefore sees thousands of candidate captions, which supplies many negatives for the ranking game. The loss is symmetric: image-to-text and text-to-image each contribute a cross-entropy term. It's often described as InfoNCE-style (Information Noise Contrastive Estimation).

CLIP trains this setup from scratch with a linear projection from each encoder into the shared space. It also learns a temperature, initialized to the equivalent of 0.07 and clipped so logits aren't scaled by more than 100.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} The temperature controls how sharply the model separates close scores.

Let $I_i$ and $T_j$ be the L2-normalized image and text embeddings. CLIP computes logits

where $t$ is a learned logit-scale parameter. You'll also see this written as division by a temperature $\tau$, where $\tau = 1 / \exp(t)$. The symmetric objective is:

The first equation asks: "Given image $i$, what's the probability that caption $i$ is the right one?" It compares the matching score $\exp(z_{ii})$ with every caption score in row $i$. The text-to-image equation does the same for column $i$.

The final loss averages both directions. Both contribute gradients, but an average can still hide a regression in one direction, so evaluate image-to-text and text-to-image retrieval separately. Diagonal labels assume the batch's pairs are aligned; related off-diagonal pairs may be false negatives.

The next implementation keeps the same symmetric pattern with a fixed temperature. It leaves out distributed batching and the learned scale so the matrix and its output stay inspectable.

import math

def normalize(vector: list[float]) -> list[float]:
    if not vector or not all(math.isfinite(value) for value in vector):
        raise ValueError("expected a nonempty finite vector")
    length = math.hypot(*vector)
    if length == 0 or not math.isfinite(length):
        raise ValueError("vector norm must be finite and nonzero")
    return [value / length for value in vector]

def dot(left: list[float], right: list[float]) -> float:
    return sum(a * b for a, b in zip(left, right, strict=True))

def cross_entropy(logits: list[float], target: int) -> float:
    if not logits or not all(math.isfinite(x) for x in logits):
        raise ValueError("logits must be nonempty and finite")
    if type(target) is not int or not 0 <= target < len(logits):
        raise ValueError("target is out of range")
    max_logit = max(logits)
    return math.log(sum(math.exp(x - max_logit) for x in logits)) - (logits[target] - max_logit)

def clip_loss(
    image_embeds: list[list[float]],
    text_embeds: list[list[float]],
    temperature: float = 0.2,
) -> tuple[float, list[list[float]]]:
    if not image_embeds or len(image_embeds) != len(text_embeds):
        raise ValueError("require equal nonempty batches of paired embeddings")
    if not math.isfinite(temperature) or temperature <= 0:
        raise ValueError("temperature must be finite and positive")
    if len({len(v) for v in image_embeds + text_embeds}) != 1:
        raise ValueError("all embeddings must share one dimension")
    images = [normalize(vector) for vector in image_embeds]
    texts = [normalize(vector) for vector in text_embeds]
    logits = [
        [dot(image, text) / temperature for text in texts]
        for image in images
    ]

    loss_i2t = sum(cross_entropy(row, i) for i, row in enumerate(logits)) / len(logits)
    columns = [list(column) for column in zip(*logits, strict=True)]
    loss_t2i = sum(cross_entropy(column, i) for i, column in enumerate(columns)) / len(columns)
    return (loss_i2t + loss_t2i) / 2, logits

image_embeds = [
    [0.95, 0.05, 0.00],
    [0.05, 0.92, 0.03],
    [0.00, 0.07, 0.94],
]
text_embeds = [
    [0.91, 0.08, 0.01],
    [0.06, 0.89, 0.04],
    [0.02, 0.09, 0.90],
]

loss, logits = clip_loss(image_embeds, text_embeds)
print("logit matrix:")
for row in logits:
    print([round(value, 2) for value in row])
print(f"symmetric_loss: {loss:.3f}")

logit matrix:
[5.0, 0.6, 0.14]
[0.71, 5.0, 0.66]
[0.09, 0.59, 5.0]
symmetric_loss: 0.022

Read the output as a matrix, not as three independent scores. The diagonal entries are all 5.0, while most off-diagonal entries are close to zero, so the paired rows and columns win. A random or poorly aligned batch would look flatter and produce a higher loss.

That matrix also explains the systems cost. Softmax-based cross-entropy needs normalization over each row and column of the effective batch. The simple implementation materializes $N \times N$ logits; an optimized one can accumulate normalization in blocks instead. Large distributed batches still require embedding/gradient communication. CLIP used 32,768 pairs, supplying many in-batch negatives.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

An off-diagonal pair is treated as negative, but it isn't automatically a clean semantic negative. Two photographs of the same rack hazard can be useful matches even when they arrived with different captions. Before scaling the batch, audit for those collisions so the loss doesn't push related examples apart.

from dataclasses import dataclass

@dataclass(frozen=True)
class Pair:
    image_id: str
    caption: str
    concept: str

pairs = [
    Pair("rack-front", "blocked vent on rack door", "blocked-airflow"),
    Pair("rack-side", "airflow blocked by loose cabling", "blocked-airflow"),
    Pair("asset-tag", "barcode obscured by cable tie", "blocked-barcode"),
]

potential_false_negatives = [
    (left.image_id, right.image_id, left.concept)
    for index, left in enumerate(pairs)
    for right in pairs[index + 1:]
    if left.concept == right.concept
]

print("pairs_treated_as_negative_but_related:", potential_false_negatives)

pairs_treated_as_negative_but_related: [('rack-front', 'rack-side', 'blocked-airflow')]

Training data

The original paper names its constructed corpus WebImageText (WIT). It contains 400 million (image, text) pairs collected from the internet, with search queries chosen to cover a broad vocabulary rather than a closed label set.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

Unlike ImageNet-style datasets built around manual class labels, CLIP learns from natural language attached to images on the web. That supervision is broader and noisier. A pair of photos showing the same blocked vent can still become in-batch negatives.

For a domain dataset, record source rights, collection purpose, caption provenance, and retention rules. Split near-duplicates and related views by asset/site/time before evaluating; otherwise the same rack can leak into train and test. Captions may omit hazards or contain asset identifiers that let the model memorize shortcuts. Audit those shortcuts and stratify evaluation by camera, site, lighting, and rare defects.

Zero-shot classification

Zero-shot classification reuses the same ranking interface. Instead of a final layer with fixed class weights, the model compares one image embedding with text descriptions of the candidate classes.

Suppose a data-center rack photo must land in one of three categories. Before looking at the result, predict what the highest similarity will tell you, and what it still won't tell you:

These invented scores explain ranking, not a pretrained model's performance. The highest score is the top candidate, not a verdict. Calibrate a score/margin policy on development data, then evaluate it on a separate held-out set. A softmax over candidate classes is conditional on that class list, not a calibrated hazard probability or an open-set detector.

The same mechanism produced a striking research result. The CLIP paper's best model, ViT-L/14@336px, reaches 76.2% zero-shot top-1 on ImageNet and matches the original ResNet-50 without using ImageNet's 1.28 million crowd-labeled training examples.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} A data-center prompt such as "cable label partially obscured by a tie" can be scored the same way, but reliability on your racks still has to be measured.

The function below reuses normalize and dot from training-objective.py. It scores hand-written vectors, not encoded rack photos. It returns both cosine similarities and candidate-relative softmax weights. No model weights are downloaded or evaluated in these exercises.

import json

def softmax(values: list[float]) -> list[float]:
    if not values or not all(math.isfinite(x) for x in values):
        raise ValueError("softmax requires nonempty finite values")
    max_value = max(values)
    exp_values = [math.exp(value - max_value) for value in values]
    total = sum(exp_values)
    return [value / total for value in exp_values]

def zero_shot_classify(
    image_embed: list[float],
    class_names: list[str],
    text_embeds: dict[str, list[float]],
    logit_scale: float = 12.0,
) -> dict[str, object]:
    if (not class_names or any(not name.strip() for name in class_names)
            or len(set(class_names)) != len(class_names)):
        raise ValueError("class names must be nonempty and unique")
    if not math.isfinite(logit_scale) or logit_scale <= 0:
        raise ValueError("logit scale must be finite and positive")
    image = normalize(image_embed)
    prompts = [f"an image showing {name}" for name in class_names]
    similarities = [dot(image, normalize(text_embeds[prompt])) for prompt in prompts]
    logits = [logit_scale * similarity for similarity in similarities]
    probs = softmax(logits)
    best_index = max(range(len(probs)), key=probs.__getitem__)
    return {
        "predicted_class": class_names[best_index],
        "cosines": dict(zip(class_names, (round(x, 3) for x in similarities), strict=True)),
        "relative_softmax": {
            class_name: round(probability, 3)
            for class_name, probability in zip(class_names, probs, strict=True)
        },
    }

text_embeds = {
    "an image showing rack hazard": [0.92, 0.20, 0.05],
    "an image showing safe rack": [0.18, 0.88, 0.08],
    "an image showing blocked asset tag": [0.48, 0.31, 0.81],
}

result = zero_shot_classify(
    image_embed=[0.89, 0.24, 0.08],
    class_names=["rack hazard", "safe rack", "blocked asset tag"],
    text_embeds=text_embeds,
)
print(json.dumps(result, indent=2))
print("singleton softmax:", softmax([-10.0]))  # 1.0 even for a poor match

# An invented norm difference reverses raw-dot ranking, not cosine ranking.
probe_image = [1.0, 0.0]
probe_features = {"caption A": [0.8, 0.6], "caption B": [9.0, 12.0]}
raw_scores = {name: dot(probe_image, vector) for name, vector in probe_features.items()}
cosine_scores = {name: round(dot(normalize(probe_image), normalize(vector)), 3)
                 for name, vector in probe_features.items()}
assert max(raw_scores, key=raw_scores.__getitem__) == "caption B"
assert max(cosine_scores, key=cosine_scores.__getitem__) == "caption A"
print("raw dot:", raw_scores)
print("normalized cosine:", cosine_scores)

{
  "predicted_class": "rack hazard",
  "cosines": {
    "rack hazard": 0.998,
    "safe rack": 0.453,
    "blocked asset tag": 0.618
  },
  "relative_softmax": {
    "rack hazard": 0.988,
    "safe rack": 0.001,
    "blocked asset tag": 0.01
  }
}
singleton softmax: [1.0]
raw dot: {'caption A': 0.8, 'caption B': 9.0}
normalized cosine: {'caption A': 0.8, 'caption B': 0.6}

The second probe isolates the API trap. Caption B's feature norm is 15, so raw dot product rewards its magnitude and picks it over A. Normalizing removes that magnitude advantage and ranks A's 0.8 cosine above B's 0.6. These are invented features, not caption-quality measurements. Bind normalization and the index metric to the generation. Inner product and cosine rank unit vectors identically; unnormalized inner product can change their order as it did here.

Prompt engineering matters

The words around a class change the text vector. For a rack photo, compare a bare rack hazard label with "a photo of a rack hazard." The latter matches the sentence-like text CLIP saw during pre-training; on ImageNet, that template alone added 1.3 percentage points of top-1 accuracy.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

One wording can still be brittle. CLIP can encode several templates, average their normalized text embeddings, and cache one class vector. The original paper ensembled 80 context prompts and gained another 3.5 percentage points over the single default template; prompt engineering plus ensembling improved ImageNet accuracy by almost 5 percentage points.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

The code makes that averaging visible with a hardcoded list of prompt vectors. No single phrasing gets to decide the class representation by itself.

def mean_vector(vectors: list[list[float]]) -> list[float]:
    if not vectors or len({len(v) for v in vectors}) != 1:
        raise ValueError("require nonempty equal-width vectors")
    return [
        sum(vector[i] for vector in vectors) / len(vectors)
        for i in range(len(vectors[0]))
    ]

prompt_vectors = {
    "a photo of a rack hazard": [0.92, 0.20, 0.05],
    "a picture of a rack hazard": [0.88, 0.24, 0.07],
    "an image showing rack hazard": [0.86, 0.28, 0.08],
    "a rack hazard in the wild": [0.82, 0.31, 0.12],
}

ensemble = normalize(mean_vector([
    normalize(vector)
    for vector in prompt_vectors.values()
]))

print(json.dumps({
    "class": "rack hazard",
    "templates": len(prompt_vectors),
    "ensemble_embedding": [round(value, 3) for value in ensemble],
}, indent=2))

{
  "class": "rack hazard",
  "templates": 4,
  "ensemble_embedding": [
    0.955,
    0.284,
    0.088
  ]
}

Run the ensemble cell after the classification cell. Averaging may reduce wording sensitivity; it doesn't guarantee better transfer. Opposing template vectors can even cancel to a zero vector, which normalization should reject. The next policy illustrates threshold mechanics with invented cosine scores and thresholds, not a calibration experiment. Tune thresholds on development data and measure risk/coverage on a separate test set.

import math

def decide(scores: dict[str, float], min_score: float, min_margin: float) -> str:
    if len(scores) < 2 or any(not math.isfinite(v) or not -1 <= v <= 1 for v in scores.values()):
        raise ValueError("require at least two finite cosine scores")
    if not -1 <= min_score <= 1 or not 0 <= min_margin <= 2:
        raise ValueError("invalid cosine threshold or margin")
    ranked = sorted(scores.items(), key=lambda item: item[1], reverse=True)
    (best_label, best_score), (_, second_score) = ranked[:2]
    if best_score == second_score or best_score < min_score or best_score - second_score < min_margin:
        return "review"
    return best_label

cases = {
    "clear_hazard": {"rack_hazard": 0.87, "safe_rack": 0.12, "tag_blocked": 0.31},
    "ambiguous_panel": {"rack_hazard": 0.56, "safe_rack": 0.53, "tag_blocked": 0.18},
}

for name, scores in cases.items():
    print(name, decide(scores, min_score=0.60, min_margin=0.10))

clear_hazard rack_hazard
ambiguous_panel review

Limitations of CLIP

Here is the boundary to test before wiring the score into an action: can one pooled similarity tell you how many vents are blocked or where each one is? No. Standard CLIP inference exposes a pooled image representation optimized for image-text matching, not calibrated counts or boxes. For counting and grounded localization, validate a detector such as Grounding DINO^{[3]Reference 3Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.https://arxiv.org/abs/2303.05499} or a grounding-aware VLM instead of interpreting global similarity as a region prediction.

That missing output is the key boundary. Zero-shot scoring compares one pooled image representation with one text representation. It returns a match score, not region coordinates or count evidence, so a concept-level hit can still come from the wrong component for an action that needs spatial proof.

Ask CLIP to "describe this image" and it returns similarity scores, not a sentence. A dual encoder has no language-generation head. Use a generative VLM such as LLaVA (Large Language-and-Vision Assistant) for captioning, or keep CLIP on retrieval and add a separate LLM for generation.

The same pooled contract can confuse relationships. Include paired probes such as "a red toolbox beside a blue safety cone" versus "a blue toolbox beside a red safety cone" in your inspection evaluation. These are proposed tests, not measured outcomes here. Grounding DINO is a separate open-set detector, not a drop-in CLIP detection head; boxes still need relation-specific evaluation and don't guarantee spatial reasoning.^{[3]Reference 3Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.https://arxiv.org/abs/2303.05499}

Counting needs its own evidence path. If a team maps the top similarity score to "three rack hazards" and a labeled audit finds five, the score has been asked to do count-supervised work it never outputs. Use region-based models or an explicit counting module instead.

Printed text is another boundary. CLIP learns useful OCR features on digitally rendered words, which are common in its pre-training data, but the paper reports highly variable transfer across domains and formats.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} Zero-shot CLIP reaches 88% on MNIST and is beaten there by logistic regression on raw pixels; full-number SVHN is only 51%, with failures on repeated characters, blur, and low resolution.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} A pooled similarity score isn't a document-OCR interface, so route rack labels through a specialized OCR stage.

Finally, web-crawled supervision carries social bias. CLIP can associate visual concepts with stereotypes.^{[4]Reference 4Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications.https://arxiv.org/abs/2108.02818} Audit embeddings on representative data before deployment, and treat debiasing or domain-specific fine-tuning as separate interventions rather than assuming a prompt fixes the problem.

Vision encoder architectures

The pooled vector is the wrong contract for counting vents or drawing boxes. To understand what a generative VLM can preserve, look one step earlier. A ViT emits a sequence of patch features before pooling; a CNN tower emits a spatial feature map that a connector can flatten into tokens. That feature count becomes a serving input even when CLIP later reduces it to one vector.

Vision Transformer (ViT)[5]Reference 5An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929

Many published VLMs use Vision Transformers (ViT), while earlier systems used Convolutional Neural Networks (CNNs) such as ResNet. ViT treats an image as a sequence of patches and runs a Transformer encoder over them, much like Large Language Models (LLMs) process text tokens. Each patch becomes an embedding, and the stack returns contextualized visual tokens. CLIP pools those tokens into one image vector; a generative VLM may project many of them into the language prefix.

Patch tokenization

Before you count, predict the shape: a 224×224 image with 16×16 patches has 14 patches along each side. That gives $14 \times 14 = 196$ visual tokens, plus a special [CLS] token used by many implementations to aggregate the global image representation.

Patch size is a token-budget choice. The figure uses 16x16 patches, while the high-resolution calculation below uses 14x14 patches. Compute the budget from the actual tower instead of copying a memorized number.

def patch_tokens(height: int, width: int, patch_size: int, crops: int = 1) -> int:
    if any(type(x) is not int or x <= 0 for x in (height, width, patch_size, crops)):
        raise ValueError("dimensions, patch size, and crops must be positive integers")
    if height % patch_size or width % patch_size:
        raise ValueError("this example requires exact patches; specify resize/padding separately")
    return (height // patch_size) * (width // patch_size) * crops

for label, size, crops in [("thumbnail", 224, 1), ("page_crop", 336, 1), ("tiled_page", 336, 4)]:
    print(label, patch_tokens(size, size, patch_size=14, crops=crops))

thumbnail 256
page_crop 576
tiled_page 2304

The arithmetic tells us how many tokens exist. Architecture determines how those tokens acquire context. The comparison below keeps that distinction visible before we choose a connector.

ViT vs. ResNet

SigLIP[6]Reference 6Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343

Standard CLIP pays for a softmax over image-text candidates in the effective batch. Original CLIP used a batch size of 32,768 in its distributed training run; smaller implementations can still train, but they change the negative-diversity and accuracy trade-off.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

SigLIP (Sigmoid Loss for Language-Image Pre-training) changes the question. Rather than asking "Which of these 32,000 captions is the best match?", it asks "Is this specific image and caption a good match, yes or no?" It replaces the global softmax competition with a pairwise sigmoid loss:

Here $N$ is the batch size, $z_{ij} = +1$ for matching pairs and $z_{ij} = -1$ otherwise. This $z$ is a label, unlike CLIP's earlier logit notation. $v_i$ and $u_j$ are normalized embeddings; $t>0$ is a learned logit scale (inverse temperature), $b$ a learned bias, and $\sigma$ the logistic sigmoid.

The paper parametrizes $t=\exp(t_0)$, initializes $t_0=\log 10$, and initializes $b=-10$. This favors negative predictions initially, matching a batch dominated by mismatches.^{[6]Reference 6Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343} Algorithm 1 sums all $N^2$ pair losses and divides by $N$, not $N^2$. Averaging every pair instead would shrink the loss and gradients by another factor of $N$.

Advantages over CLIP

• Decoupled computation: No global normalization is needed. Each pair is scored independently, allowing the similarity matrix to be computed in chunks across devices.^{[6]Reference 6Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}
• Efficiency: The sigmoid loss removes global softmax normalization. The SigLIP paper reports a large gap versus softmax when the batch is smaller than 16k, a smaller gap as the batch grows, and diminishing returns past a more reasonable 32k. Pushing SigLiT to a 1 million batch was possible; it didn't keep helping.^{[6]Reference 6Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}
• Memory and partitioning: The loss is easier to chunk across devices because it doesn't require a single global softmax over the full batch.

SigLIP's pairwise loss changes the computation contract: positive and negative pair terms can be evaluated without a batch-wide probability distribution. That makes partitioning easier, but it doesn't decide what accelerator, data volume, or accuracy a data-center matcher will need.

import math

def softplus(value: float) -> float:
    return max(value, 0.0) + math.log1p(math.exp(-abs(value)))

def pair_loss(score: float, is_match: bool) -> float:
    if not math.isfinite(score) or type(is_match) is not bool:
        raise ValueError("require a finite logit and a boolean match label")
    label = 1.0 if is_match else -1.0
    return softplus(-label * score)

scores = [
    ("same_asset", 3.2, True),
    ("wrong_caption", -2.1, False),
    ("hard_negative", 0.4, False),
]

for name, score, match in scores:
    print(name, round(pair_loss(score, match), 3))

# Four pairs, two examples. At zero logits each pair costs log(2).
batch_logits = [[0.0, 0.0], [0.0, 0.0]]
n = len(batch_logits)
total = sum(pair_loss(score, i == j)
            for i, row in enumerate(batch_logits) for j, score in enumerate(row))
print("paper reduction:", round(total / n, 3))
print("pair mean:", round(total / (n * n), 3))

same_asset 0.04
wrong_caption 0.116
hard_negative 0.913
paper reduction: 1.386
pair mean: 0.693

Why SigLIP matters in newer stacks

Some published VLMs use SigLIP-family towers when moderate-batch training or partitioned pair scoring matters. SigLIP 2 adds caption-based pretraining, self-distillation, masked prediction, and online data curation; its paper reports better transfer than original SigLIP across model scales when used as a VLM vision backbone.^{[7]Reference 7SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} Its NaFlex variants support multiple resolutions while preserving native aspect ratio. Standard fixed-resolution checkpoints don't acquire that behavior simply by being SigLIP 2: NaFlex needs its matching vision implementation and adapted preprocessing.^{[8]Reference 8SigLIP 2: Official Checkpoint and Implementation Guidehttps://github.com/google-research/big_vision/blob/0127fb6b337ee2a27bf4e54dea79cff176527356/big_vision/configs/proj/image_text/README_siglip2.md}

For a wide rack panorama, choose and validate a supported checkpoint/preprocessing pair before counting patches. “Native aspect ratio” doesn't mean the original image keeps every pixel or that an arbitrary token budget is safe. The foundation remains image-text alignment, and sigmoid match scores still need target-domain calibration rather than being treated as hazard probabilities.

EVA-CLIP: scaling vision backbones with masked pretraining[9]Reference 9EVA-CLIP: Improved Training Techniques for CLIP at Scalehttps://arxiv.org/abs/2303.15389

Training a large Vision Transformer directly from scratch with contrastive loss is expensive, and scaling can become unstable. EVA-CLIP initializes its vision tower from an EVA masked image modeling (MIM) model, then uses that representation to make contrastive scaling more stable and efficient.^{[9]Reference 9EVA-CLIP: Improved Training Techniques for CLIP at Scalehttps://arxiv.org/abs/2303.15389} The tower therefore starts with features learned by reconstructing masked visual features from a teacher, rather than learning every visual pattern from image-text pairs alone.

The hybrid recipe also changes the sample-efficiency result. The paper's EVA-02-CLIP-L/14+ has about 430M total model parameters and reaches 80.4% zero-shot top-1 on ImageNet-1K; EVA-02-CLIP-E/14+ has 5.0B total parameters and reaches 82.0%.^{[9]Reference 9EVA-CLIP: Improved Training Techniques for CLIP at Scalehttps://arxiv.org/abs/2303.15389} Those totals include both towers, not only the vision encoder. The results describe ImageNet evaluation, not a guarantee that a data-center tower will read small asset tags.

Modern VLMs: from alignment to generation

The similarity score has reached its boundary. CLIP and SigLIP can say whether an image and caption belong together, but a technician asking "what's covering the barcode?" needs generated text and perhaps a box.

That question introduces fusion: the point where visual evidence and text interact to produce an answer. Before choosing a connector, predict the serving trade-off. Passing every patch to the language model preserves detail but lengthens its input; compressing patches protects the input budget but may discard the barcode.

Three published architectures illustrate the connector patterns. These dated examples aren't a current leaderboard or a claim that newer checkpoints retain the same input limits:

• simple projectors that map vision features into the LLM space
• query-based compressors that reduce many visual patches into a fixed token budget
• resamplers and adapters that preserve more spatial detail for high-resolution inputs

These patterns are enough to reason about token flow without guessing at proprietary internals.

LLaVA: the straightforward projector path[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485

LLaVA is the clearest example of projecting visual features into a language model. Original LLaVA takes a pretrained CLIP ViT-L/14 encoder and Vicuna (an instruction-tuned LLaMA chat model), then learns a linear projection matrix $W$ that maps encoder grid features into the LLM's word-embedding dimension.^{[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} The projector changes representation space; it doesn't decide whether a barcode is present.

The original recipe separates two learning problems:^{[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}

1. Feature alignment: freeze the vision encoder and LLM, train only $W$ on a filtered slice of Conceptual Captions 3M (595K image-text pairs after filtering).
2. Visual instruction tuning: fine-tune the projector and LLM on 158K GPT-4-generated multimodal instructions.

The data-generation teacher was text-only: GPT-4 received captions and object-box annotations, not image pixels. It synthesized conversations about that symbolic evidence.^{[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} Missing or incorrect annotations can therefore reach the training answers; fluent synthetic instructions still need data-quality checks.

LLaVA-1.5 keeps that two-stage shape and changes two serving-relevant pieces: a two-layer MLP connector replaces the linear layer, and CLIP-ViT-L-336px replaces the 224px input. That resolution yields a $24 \times 24$ grid, or 576 visual prefix tokens per image.^{[11]Reference 11Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744} The pattern stays simple: a pretrained vision encoder, an LLM, and a learned projector can support multimodal instruction following without a new vision backbone.

Common multimodal training ladder

The LLaVA split illustrates a common training ladder. This staged recipe separates representation learning, connector alignment, and answer behavior. The rows below are common options, not a required sequence for every VLM; which components are frozen or trained jointly depends on the architecture and recipe.

CLIP and SigLIP mostly cover the first row.^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020[6]Reference 6Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343} LLaVA makes the second and third rows explicit with feature alignment followed by visual instruction tuning.^{[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} A chat-oriented system needs training beyond contrastive alignment because matching doesn't teach long-form answer behavior.

Use the observed failure to locate the missing stage:

• contrastive pretraining teaches matching
• instruction tuning teaches how to answer
• post-training teaches which answers are preferred or allowed

A response-format failure suggests checking instruction data and the chat template. A safety failure needs policy enforcement and evaluation, with post-training as one possible intervention. Neither symptom uniquely identifies a training stage: bad features, prompts, or data can also cause failures. BLIP-2's Q-Former addresses a separate question: how much visual information reaches the language path?

BLIP-2 and the Q-Former: compress before the LLM[12]Reference 12BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597

BLIP-2 (Bootstrapping Language-Image Pre-training, version 2) addresses the token bottleneck directly. Instead of forwarding every visual feature into the LLM, it introduces a Q-Former that learns a fixed set of query vectors. The queries pull task-relevant information from a frozen vision encoder.

The published setup uses 32 queries of dimension 768. That $32 \times 768$ bottleneck is much smaller than frozen ViT-L/14 features ($257 \times 1024$, including the class token), and the query count stays fixed when input resolution rises.^{[12]Reference 12BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} The paper reports that BLIP-2 outperforms Flamingo-80B by 8.7 points on zero-shot VQAv2 while using $54\times$ fewer trainable parameters.^{[12]Reference 12BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} That's a research comparison, not a serving SLA for rack photos.

The serving consequence is straightforward: tokens admitted to the language path affect prefill and cache cost. A smaller fixed budget wins only if it preserves task-critical evidence. If 32 queries drop characters on an asset tag, lower prefix cost is the wrong result.

Flamingo-style resampling: preserve more context without exploding token count[13]Reference 13Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198

Flamingo takes a different route to fusion. It first compresses variable-length visual features with a Perceiver Resampler, then lets language tokens attend to that compact bank through gated cross-attention blocks inserted into a frozen LM.^{[13]Reference 13Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198}

The published resampler emits 64 visual tokens per image/video item, independent of its resolution or frame count.^{[13]Reference 13Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} Multiple items still create multiple banks. Flamingo uses a frozen contrastively pretrained NFNet-F6, not a CLIP ViT. Its cross-attention mask directly exposes the most recent preceding image/video to a text token; older visual information can flow indirectly through text self-attention. A $\tanh$ gate initialized at zero initially preserves the frozen LM's behavior.^{[13]Reference 13Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198}

Predict the workload where this boundary helps most. A single low-resolution photo may not justify cross-attention machinery, but the trade-off changes when a prompt contains:

• multiple images appear in one prompt
• video or long visual sequences are involved
• the language model needs to revisit visual evidence later in the answer

Resamplers and cross-attention can preserve a revisitable visual bank, but they add more moving parts than the simple LLaVA prefix. That complexity belongs in the design only when multiple images, long visual sequences, or repeated visual access pay it back.

Qwen-VL: grounding and text reading beyond caption matching[14]Reference 14Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondhttps://arxiv.org/abs/2308.12966

Qwen-VL (Qwen Vision-Language) shows what changes when grounding and text reading are first-class tasks. Its paper describes a visual receptor, an input-output interface, and a multi-stage training pipeline for OCR-heavy and box-grounded work in addition to generic image understanding.^{[14]Reference 14Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondhttps://arxiv.org/abs/2308.12966}

Those tasks need spatial signal. A tiny asset tag or a box around a vent can disappear when a high-resolution image is reduced to one pooled representation.

High-resolution token math

For a separate, generic budget example, suppose the tower uses 14×14 patches. One 336×336 crop becomes $24 \times 24 = 576$ patch tokens; four crops become 2,304. This isn't Qwen-VL's exact preprocessing contract. More crops preserve more pixel detail, but quality needn't improve: seams, repeated regions, and missing global context can introduce errors. The projected language prefix still gets longer.

In a projected-prefix VLM, those admitted visual tokens cost twice. They lengthen prefill, and they enlarge the KV cache that must remain resident during decode. PagedAttention can pack and recycle that memory more efficiently, but it doesn't reduce the prefix token count.^{[15]Reference 15Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Cross-attention designs keep a separate visual bank, so measure their memory and latency path separately.

In a projector path, the connector decides how many encoder features become autoregressive prefix tokens. Other architectures expose a separate cross-attention memory bank instead. The comparison below uses raw tiles and a fixed-size compression path to count tokens; it doesn't claim that compression preserves enough OCR evidence.

def prefix_tokens(text_tokens: int, crops: int, patch_tokens_per_crop: int, compressed_tokens: int | None) -> int:
    if (type(text_tokens) is not int or text_tokens < 0
            or any(type(x) is not int or x <= 0 for x in (crops, patch_tokens_per_crop))):
        raise ValueError("invalid text, crop, or patch count")
    visual_tokens = crops * patch_tokens_per_crop
    if compressed_tokens is not None:
        if type(compressed_tokens) is not int or not 0 < compressed_tokens <= visual_tokens:
            raise ValueError("compression must retain a positive bounded token count")
        visual_tokens = compressed_tokens
    return text_tokens + visual_tokens

raw_prefix = prefix_tokens(text_tokens=180, crops=4, patch_tokens_per_crop=576, compressed_tokens=None)
compressed_prefix = prefix_tokens(text_tokens=180, crops=4, patch_tokens_per_crop=576, compressed_tokens=64)

print("raw_prefix_tokens:", raw_prefix)
print("compressed_prefix_tokens:", compressed_prefix)
print("tokens_avoided_if_quality_holds:", raw_prefix - compressed_prefix)

raw_prefix_tokens: 2484
compressed_prefix_tokens: 244
tokens_avoided_if_quality_holds: 2240

The 64-token case is a hypothetical prefix compressor over the complete crop set, not Flamingo's cross-attention architecture. These totals omit image separators, chat framing, and generated output; reserve those separately before admitting the request. If a compressor emits 64 tokens per crop instead, four crops contribute 256, not 64.

When detail is expendable, resize. When small objects or text matter, tile or crop the image and retain crop coordinates so OCR and boxes can map back to the original image. Compression can reduce language-model cost, but a fixed query count doesn't make arbitrary input resolutions compatible with a pretrained vision tower.

Whichever path you pick, measure the evidence path with the budget. Check resolution, tiling, OCR, connector compression, and the model's measured failure slices together. BLIP-2's Q-Former and Flamingo's resampler show ways to bound or mediate visual features before language reasoning, but neither removes the need to test the asset-tag slice.^{[12]Reference 12BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597[13]Reference 13Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198}

Adaptive pruning and merging under a latency budget

A fixed visual-token budget makes capacity predictable, but it spends the same allowance on a blank panel and a dense wiring diagram. An adaptive reducer estimates which tokens carry useful information for each image, then drops or merges the rest.

Pruning and merging are different operations:

• Pruning removes low-importance tokens. It's cheap, but discarded evidence can't be recovered.
• Merging combines redundant tokens into retained representatives. It can preserve more aggregate context, but similarity mistakes can blend distinct objects or text regions.

PruMerge is one researched example for projected-prefix VLMs.^{[16]Reference 16LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelshttps://arxiv.org/abs/2403.15388} It uses sparse attention between the vision encoder's class token and spatial tokens to select important tokens per image. It then clusters tokens by key similarity and folds related pruned tokens into retained representatives with weighted averaging. The paper evaluates it on LLaVA-1.5 and Video-LLaVA. That makes it a candidate reducer, not a guarantee for every encoder, OCR route, or domain.

The reducer still needs a hard serving contract. If a dense label panel generates more candidates than the route can afford, the cap must win:

The route should also reserve minimum coverage for task-critical evidence. A dashboard policy might keep tokens that overlap OCR boxes, small-object proposals, or selected crops before spending the remaining budget on saliency. A global spatial sample protects against an importance scorer that focuses on one object.

Placement determines which cost disappears. Reducing tokens after the vision encoder saves connector and language-model work, but it doesn't save the completed vision-encoder pass. A reducer inside the vision tower can save later encoder work, but it changes more of the model and needs separate evaluation.

Track the admitted-token distribution, reduction overhead, vision latency, prefill time, time to first token, peak memory, and quality by slice. For this inspection product, OCR exact match, small-hazard recall, and grounding IoU matter more than average VQA alone. If the hard cap would remove protected evidence, use a slower route or return review instead of silently forcing the image through the fast path.

Make that conflict visible with four invented token IDs already ranked by saliency. The warning label ranks last. Predict whether taking the first two IDs preserves the evidence needed by the technician.

def admit_tokens(ranked: list[str], protected: set[str], cap: int) -> list[str] | None:
    if type(cap) is not int or cap <= 0 or len(set(ranked)) != len(ranked):
        raise ValueError("positive cap and unique token IDs required")
    if not protected <= set(ranked):
        raise ValueError("protected evidence is absent from the candidates")
    if len(protected) > cap:
        return None  # Policy cannot satisfy both contracts on this route.
    selected = set(protected)
    for token in ranked:
        if len(selected) >= cap:
            break
        selected.add(token)
    # Keep original token order separately in the model's actual input.
    return [token for token in ranked if token in selected]

ranked = ["rack-door", "logo", "cable", "warning-text"]
protected = {"rack-door", "warning-text"}  # Trusted task policy, not model permission.
print("saliency_only:", ranked[:2])
print("protected_selection:", admit_tokens(ranked, protected, cap=2))
print("cap_one:", "review" if admit_tokens(ranked, protected, cap=1) is None else "admitted")
assert admit_tokens(ranked, protected, 2) == ["rack-door", "warning-text"]
assert admit_tokens(ranked, protected, 3) == ["rack-door", "logo", "warning-text"]
try:
    admit_tokens(ranked, {"unseen-tag"}, 2)
except ValueError as error:
    assert str(error) == "protected evidence is absent from the candidates"
else:
    raise AssertionError("missing evidence must not look like successful admission")

saliency_only: ['rack-door', 'logo']
protected_selection: ['rack-door', 'warning-text']
cap_one: review

Both two-token results meet the cap, but only the protected selection includes the warning. At cap one, no valid fast-path subset exists, so the policy returns review instead of deleting required evidence. This fixture selects IDs; it doesn't detect text or implement PruMerge. A real reducer must preserve selected positions, spatial relationships, and the trained input contract, then measure whether its protected regions actually retain legible evidence.

Connector design controls a serving trade-off

The component between the vision encoder and the LLM answers two serving questions: what visual evidence reaches generation, and how many visual tokens the language path must handle. A connector can save context and still fail the barcode task if it compresses the wrong region.

Compare four connector contracts:

1. Linear projection: original LLaVA's $W$. Changes feature width without changing token count.^{[10]Reference 10Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}
2. MLP projector: LLaVA-1.5's two-layer connector. Adds a nonlinear mapping; its benefit is an empirical model/training result, not a universal guarantee.^{[11]Reference 11Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744}
3. Q-Former / query transformer: compresses many visual features into a fixed learned token set.^{[12]Reference 12BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597}
4. Convolutional or locality-aware abstractors: preserve more spatial structure before handing features to the LLM.^{[17]Reference 17Honeybee: Locality-enhanced Projector for Multimodal LLMhttps://arxiv.org/abs/2312.06742}

An overcompressed connector can remove OCR or grounding evidence. A large or expensive connector can increase latency or language-prefix cost. Measure both outcomes by task slice.

There isn't one best adapter. Choose against the task you have to protect:

• raw simplicity
• token efficiency
• spatial fidelity
• video or multi-image support

Designing around unpublished internals

Closed vendors may publish capability demos and benchmark numbers without enough architecture detail to justify an internals claim. Treat that missing detail as a boundary, not an invitation to guess.

Treat the model as an evaluated dependency. Measure supported input resolution, visual-token billing if exposed, latency by image count, OCR and grounding slices, refusal behavior, and output contract. Don't infer a projector type, token budget, or training curriculum unless the provider publishes it.

Evaluation

The connector decision is only a hypothesis until evaluation shows which evidence survives. A VLM needs more than one benchmark because retrieval, captioning, OCR, grounding, and reasoning stress different abilities.

Read the table as separate failure probes. A CLIP-like model can retrieve well and still fail counting. Reference-caption overlap can look fine while spatial reasoning is weak, and OCR-heavy tasks expose errors that general image description hides.

For the inspection product, keep these capability slices separate and set a pass bar for each one. A strong aggregate score can't rescue a route that loses asset-tag characters.

The small evaluation below computes metrics from explicit synthetic predictions. It doesn't call a vision model. Retrieval recall counts relevant IDs present in the top ten, OCR uses exact string equality, and box IoU is intersection area divided by union area in normalized image coordinates. One OCR character is wrong despite a relevant retrieved photo and an overlapping box.

import math

def box_iou(left: tuple, right: tuple) -> float:
    for box in (left, right):
        if (len(box) != 4 or any(not math.isfinite(x) or not 0 <= x <= 1 for x in box)
                or box[0] >= box[2] or box[1] >= box[3]):
            raise ValueError("expected normalized xyxy boxes with positive area")
    width = max(0, min(left[2], right[2]) - max(left[0], right[0]))
    height = max(0, min(left[3], right[3]) - max(left[1], right[1]))
    intersection = width * height
    area = lambda b: (b[2] - b[0]) * (b[3] - b[1])
    return intersection / (area(left) + area(right) - intersection)

def evaluate(rows: list[dict]) -> dict[str, float]:
    if not rows or any(not row["relevant"] for row in rows):
        raise ValueError("this fixture requires rows with relevant-image labels")
    return {
        "retrieval_recall_at_10": sum(
            len(set(row["retrieved"][:10]) & set(row["relevant"])) / len(set(row["relevant"]))
            for row in rows) / len(rows),
        "ocr_exact_match": sum(row["ocr"] == row["tag"] for row in rows) / len(rows),
        "mean_box_iou": sum(box_iou(row["box"], row["truth_box"]) for row in rows) / len(rows),
    }

rows = [
    {"relevant": ["rack-a"], "retrieved": ["rack-a"], "tag": "A-17", "ocr": "A-17",
     "truth_box": (0, 0, 1, 1), "box": (0, 0, 1, 1)},
    {"relevant": ["rack-b"], "retrieved": ["other", "rack-b"], "tag": "B-08", "ocr": "B-O8",
     "truth_box": (0, 0, 1, 1), "box": (0, 0, 1, 1)},
    {"relevant": ["rack-c"], "retrieved": ["rack-c"], "tag": "C-42", "ocr": "C-42",
     "truth_box": (0, 0, 1, 1), "box": (0, 0, 0.5, 0.5)},
]
metrics = evaluate(rows)
thresholds = {"retrieval_recall_at_10": 0.90, "ocr_exact_match": 0.92, "mean_box_iou": 0.70}
failed = [name for name, value in metrics.items() if value < thresholds[name]]
print({name: round(value, 3) for name, value in metrics.items()})
print("failed_slices:", failed)

{'retrieval_recall_at_10': 1.0, 'ocr_exact_match': 0.667, 'mean_box_iou': 0.75}
failed_slices: ['ocr_exact_match']

This three-row fixture checks metric plumbing, not release readiness. Include missing-detection and no-hazard cases, false positives and false negatives, multiple boxes with explicit matching, grouped train/test splits, and uncertainty at realistic sample sizes. Mean IoU alone can hide missed hazards; OCR exact match alone can hide abstention coverage. Report both quality and review rate rather than averaging unlike metrics into one score.

Production applications

Document understanding

Document AI is a token-budget problem. Dense documents contain tiny text, tables, figures, and layout cues that don't survive aggressive resizing.

Start with page tiling or crop-based preprocessing when small regions matter. Add OCR features for text that has to be copied exactly, then send ambiguous or layout-heavy cases to a VLM. The stages preserve different kinds of evidence.

The design target isn't maximum raw resolution. Preserve the evidence that matters while keeping the visual token budget survivable.

Image search and retrieval

For image search, start with a dual encoder rather than a generative VLM. The search contract is ranking, not explaining.

A dual-encoder model like CLIP or SigLIP lets you precompute image embeddings offline, index them once, and embed only the query at request time.

This split keeps first-stage latency and cost measurable. A generative VLM earns a place after retrieval only when reranking or explanation needs its extra evidence contract and measured cost.

The running rack photo now forks by evidence type. Similarity search, OCR, generation, and grounding are different contracts, so don't send every photo through the generative VLM because one ticket asked for a caption.

The function turns that fork into a routing plan only. It doesn't execute a model or enforce authorization. Retrieval stays on the dual encoder; exact label copying can stop at OCR, while interpretation may use a VLM. Clicking a disable port requires an independently enforced action policy and human approval bound to the current target and screenshot, not a model-written approved field.

def select_path(task: str) -> list[str]:
    if task == "retrieve_similar_assets":
        return ["dual_encoder", "vector_index"]
    if task == "answer_from_label_text":
        return ["ocr", "generative_vlm", "citation_check"]
    if task == "click_disable_port":
        return ["grounding_model", "action_policy", "human_confirm"]
    raise ValueError(task)

for task in ["retrieve_similar_assets", "answer_from_label_text", "click_disable_port"]:
    print(task, "->", " + ".join(select_path(task)))

retrieve_similar_assets -> dual_encoder + vector_index
answer_from_label_text -> ocr + generative_vlm + citation_check
click_disable_port -> grounding_model + action_policy + human_confirm

Visual agents

Visual agents need more than a captioning model. They need grounding, a link between an answer and the region or coordinate an action will use.

Suppose the agent must disable a port from a screenshot. "The blue button is near the top-right" isn't an executable target. The system may have to emit:

• click (x, y)
• select the second menu item
• type into the input field beneath a label

That usually requires a grounding-aware model or a separate grounding component on top of the core VLM, plus safety checks around actions. The same boundary applies to a rack hazard: a caption can explain, but policy needs a region and a human-confirmed action.

Common pitfalls

A useful diagnosis starts with the output contract that failed. Ask what the service returned, what evidence the task required, and where those two stopped matching.

"One pooled CLIP embedding understands an image like a person does"

If a team asks one global embedding for counts, boxes, or grounded actions, the output contract is already too broad. CLIP learned global alignment, not dense spatial supervision. Use detectors, grounding-aware VLMs, or region features when localization or counting matters.

"CLIP or SigLIP can also serve my chat and captioning path"

Similarity scores are the wrong output when the product needs an open-ended explanation or grounded dialogue. Keep CLIP or SigLIP for retrieval and zero-shot scoring, then use a generative VLM for the answer path.

"Higher resolution always means better OCR"

If document quality rises a little while latency and memory spike, resolution was increased without a token, tiling, or compression plan. Measure visual token count, tile selectively, preserve the evidence that matters, and compress before the LLM when possible.

"Visual serving cost ends when the encoder finishes"

Offline profiling can look cheap while production decode concurrency collapses. The missing terms are prefill latency and KV-cache growth from large visual prefixes, so track both with encoder cost before choosing image resolution or crop count.

"Adaptive pruning makes the latency budget self-enforcing"

Typical images can stay fast while dense screenshots create token spikes or erase tiny defects. That happens when the reducer has no hard maximum, protected evidence floor, or failure route. Cap the admitted sequence, preserve task-critical regions, bucket requests by retained count, and route unsafe reductions to a slower path or review.

"Every in-batch non-match is a clean negative"

Contrastive training can underperform on visually similar assets or captions that describe the same scene in different words. Some in-batch negatives are false negatives, so curate batches carefully, use richer captions, and treat batch design as a modeling choice.

"Zero-shot prompts will hold in fine-grained domains"

Prompt wording can swing predictions or collapse specialist categories when generic templates lack domain signal. Ensemble prompts, calibrate on validation data, and fine-tune or probe when domain precision matters.

What the inspection system must prove

Leave a reviewer with evidence for each route, not one attractive aggregate score. Your implementation is ready for review when you can:

• Explain how CLIP's dual-encoder architecture aligns images and text in a shared embedding space.
• Walk through the contrastive loss logic with a concrete batch example.
• Implement zero-shot classification using prompt engineering and embedding similarity.
• Diagnose why CLIP fails on spatial reasoning, counting, or fine-grained OCR.
• Compare LLaVA, BLIP-2, and Flamingo-style designs for connecting vision encoders to language models.
• Size the visual token budget for a production VLM and choose fixed or adaptive compression with explicit latency and evidence protections.

Evaluation rubric

• Derive both directions of CLIP loss and distinguish SigLIP's sum-over-pairs divided by batch size from a pair mean.
• Reject zero/nonfinite vectors, malformed batches, and invalid token budgets; distinguish cosine scores from candidate-relative softmax weights.
• Bind the index to the paired encoders and preprocessing, and keep permissions, image content, and action approval separate.
• Compute retrieval, OCR, and grounding metrics from predictions, then explain why synthetic arithmetic isn't pretrained model evaluation.

Follow-up questions