Vision-Language Models & CLIP

LLM-powered search turned retrieved evidence into cited answers. Vision-language models add visual evidence to that same loop: images, diagrams, screenshots, document pages, and product photos become searchable and reasoned about alongside text.

Vision-language models connect images and text so systems can search, classify, and reason across both modalities. This chapter starts with CLIP-style contrastive learning, then expands to production uses and failure modes.

Imagine training a warehouse system to recognize damaged packages. You wouldn't just give it a fixed list of labels like damaged_box or intact_box. You'd show it real photos paired with descriptions: "corner crushed during transit," "shipping label torn near barcode," or "fragile sticker visible on side panel." The system learns the concept from images and language together.

Many supervised computer vision systems expose a fixed label inventory: recognizing a new category generally requires labeled examples and another training step. CLIP (Contrastive Language-Image Pre-training) showed that image-text matching at web scale can produce representations that transfer to new text-defined classes without training a new classifier head.^[1] That unlocked practical open-vocabulary retrieval and provided useful vision features for later generative systems.

Conceptual foundation

A quick refresher: embeddings and similarity

Before we dive into CLIP, recall what embeddings are: high-dimensional vectors that compress meaning into a list of numbers. A sentence like "corner crushed during transit" becomes a vector, perhaps 768 numbers long. An image of that same crushed box becomes another 768-number vector.

If the model has done its job, those two vectors point in roughly the same direction in space. We measure that alignment with cosine similarity, a score between -1 and 1. Within one trained model, a higher score ranks an image-text pair as more aligned than a lower-scoring candidate. It is not a calibrated probability and does not prove that the pair means exactly the same thing. That's the geometric heart of CLIP: it learns to rank matching image-text pairs above mismatches.

From fixed labels to open vocabulary

Imagine trying to learn a warehouse photo taxonomy from a fixed label list. You might learn that one class is damaged_box, but you wouldn't understand how damage appears across lighting, angles, materials, and carrier labels.

Now imagine learning from many real catalog and logistics images with full captions. This is how Vision-Language Models (VLMs) learn. Instead of memorizing fixed categories, they learn to align the visual concept of an object with its natural language description.

This shared image-text space allows a system to score text prompts that were not fixed training labels. Transfer still depends on whether pre-training learned the relevant visual evidence and whether the new domain resembles the data it saw. A warehouse team must validate prompts on its own cartons, lighting, and damage policy before using scores operationally.

Shared image-text space

Think of CLIP as a shared routing map for vision and language. One side starts with pixels, the other with words. They do not share raw input format, but the original model was trained on 400 million image-text pairs.^[1] Over training, matched images and captions receive higher similarity than mismatches. If its training produced useful package-damage features, it may rank a new prompt such as "torn corner on shipping box" well even when that exact label was not a training class. That is a transfer hypothesis to evaluate, not an automatic guarantee.

CLIP: Contrastive Language-Image Pre-training

Architecture^[1]

CLIP jointly trains an image encoder and text encoder to align visual and textual representations in a shared embedding space. In the original paper, the vision side was either a ResNet (Residual Network) or a ViT (Vision Transformer), while the text side was a Transformer that produced one representation for each caption.^[1] The contrastive objective pulls matching pairs together and pushes mismatches apart, which is what makes zero-shot transfer and cross-modal retrieval work.

The illustration above shows core CLIP idea: keep encoders separate, normalize both outputs, then judge whether image and text land near each other in one shared space.

Training objective

Think of this process like matching warehouse photos to their correct captions. For every image, the model must pick the correct caption out of thousands of incorrect ones.

Let's walk through a tiny batch first, because the pattern is easier to see with real numbers. Suppose you have 3 images and 3 captions:

There are $3 \times 3 = 9$ possible pairings. Only the 3 diagonal pairs (A+A, B+B, C+C) are correct. The other 6 are mismatches. CLIP's job is to make the similarity scores for the diagonal pairs as high as possible, while keeping the off-diagonal scores low.

In a real training run, the batch size is much larger (32,768 in the original paper^[1]), so each image sees thousands of negative examples. But the logic is the same: maximize the similarity of matching pairs and minimize it for non-matching pairs using a symmetric contrastive loss that's often described as InfoNCE-style (Information Noise Contrastive Estimation).

Let $I_i$ and $T_j$ be the L2-normalized image and text embeddings. CLIP computes logits

$$z_{ij} = \exp(t) \cdot I_i^\top T_j$$

where $t$ is a learned logit-scale parameter. You'll also see this written as division by a temperature $\tau$, where $\tau = 1 / \exp(t)$. The symmetric objective is:

$$\mathcal{L}_{I \rightarrow T} = -\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(z_{ii})}{\sum_{j=1}^{N}\exp(z_{ij})}$$ $$\mathcal{L}_{T \rightarrow I} = -\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(z_{ii})}{\sum_{j=1}^{N}\exp(z_{ji})}$$ $$\mathcal{L} = \frac{\mathcal{L}_{I \rightarrow T} + \mathcal{L}_{T \rightarrow I}}{2}$$

The first equation asks: "Given image $i$, what's the probability that caption $i$ is the right one?" It compares the matching score $\exp(z_{ii})$ against the sum of all scores for that image across every caption in the batch. The second equation does the same from the text side. The final loss averages both directions so neither encoder dominates.

The implementation below shows a simplified fixed-temperature version of that loss. It keeps the math readable while matching the same symmetric training pattern.

Low loss means the diagonal image-caption pairs dominate the off-diagonal mismatches. A random or poorly aligned batch would have a much flatter matrix and a higher loss.

The softmax-based cross-entropy requires the full $N \times N$ similarity matrix, which means CLIP benefits from very large batch sizes (32,768 in the original paper^[1]) so that each example sees many in-batch negatives. In practice, that pushes training toward large distributed runs with expensive cross-device synchronization.

An off-diagonal pair is treated as negative by this objective, but it is not automatically a clean semantic negative. Two photographs of the same product defect may be valid matches even if they arrived as separate labeled pairs. A batch-construction audit catches obvious collisions before the loss pushes them apart.

Training data

The success of CLIP is heavily dependent on the sheer scale and diversity of its pre-training data. The original paper doesn't release or assign a canonical public name to the full corpus. It describes training on 400 million (image, text) pairs collected from the internet.^[1]

Unlike prior vision datasets (e.g., ImageNet) that relied on manual class labels, CLIP learns from natural language supervision attached to images on the web. That means the model trains against captions and descriptive text rather than a closed vocabulary of hand-assigned categories. The result is a much broader representation of visual concepts and a much stronger zero-shot transfer story.

Zero-shot classification

CLIP enables classification without any task-specific training. Instead of a final classification layer with fixed weights, the model computes the similarity between the image and the text descriptions of possible classes.

Analogy: instead of teaching a child that "Class 42 is a zebra," you give them a dictionary. They look at a picture and find the word in the dictionary that best describes it.

Here's a concrete example. Suppose you have a warehouse photo and you want to classify it into one of three categories:

The highest score is the model's top candidate. In a real decision path, it should win only if validation has established an acceptable score or margin policy; ambiguous images should abstain or route to review. You did not retrain the model, but you still need to calibrate the decision rule.

The practical breakthrough isn't classification alone; it's a text-defined candidate vocabulary. A warehouse damage-claim system can test a prompt such as "poly mailer with barcode partially obscured by packing tape" without fitting a new label head. Whether the prompt is reliable on real claims still has to be measured.

The function below shows the standard zero-shot pattern: prompt the candidate classes in natural language, encode image and text, normalize both, then score them with a scaled dot product. This standalone version uses small hand-written vectors so the inference mechanics are visible.

Prompt engineering matters

The format of the text prompt significantly impacts performance. "A photo of a {class}" typically works better than just "{class}".^[1]

Ensembling multiple prompts can systematically improve the stability of zero-shot classifications. The snippet below takes a hardcoded list of templates as input to format the target class description in various contexts. By tokenizing and passing each variation through the text encoder and averaging the resulting embeddings, the procedure outputs a single, more stable vector representation that mitigates the variance of any individual prompt template.

Prompt ensembling reduces wording sensitivity; it does not create a deployment threshold. Add an abstention rule using held-out data, and make uncertain classifications visible to downstream policy.

Limitations of CLIP

A common deployment mistake is using CLIP's global similarity score to count inventory from warehouse shelf photos or localize damaged regions on a package. Standard CLIP inference exposes a pooled image representation optimized for image-text matching, not calibrated counts or boxes. For counting and grounded localization, validate a detector such as Grounding DINO^[2] or a grounding-aware VLM instead of interpreting global similarity as a region prediction.

CLIP was an important advance in vision-language alignment, but its contrastive objective and standard scoring interface impose specific constraints. Because zero-shot scoring compares one pooled image representation with one text representation, the output does not expose region coordinates or counting evidence. Understanding that boundary matters before using it in production systems.

No generation. CLIP can only classify or retrieve; it can't generate text captions or new images on its own.

• Symptom: You ask CLIP to "describe this image" and it gives you a similarity score instead of a sentence.
• Cause: CLIP is a dual-encoder alignment model, not a decoder. It has no language-generation head.
• Fix: Use a generative VLM like LLaVA (Large Language-and-Vision Assistant) for captioning, or use CLIP for retrieval and a separate LLM for generation.

Compositional understanding can be weak. For example, CLIP benchmarks show failures when concepts are rearranged into different relations, such as "a red box beside a blue return bin" versus "a blue box beside a red return bin."

• Symptom: Zero-shot accuracy drops sharply on tasks that require understanding spatial relationships between multiple objects.
• Cause: The training objective only asks "does this caption match this image overall?" It doesn't supervise exact object positions or relations.
• Fix: For spatial reasoning, use grounding-aware models or add a specialized detection head like Grounding DINO.

Fine-grained tasks need separate validation. Standard pooled scoring is a poor contract for counting ("how many boxes") or exact spatial reasoning because it does not output region-level evidence.

• Symptom: A team maps a top similarity score to "three people," but a labeled audit shows five.
• Cause: The deployed scoring path returns global alignment rather than count-supervised region predictions.
• Fix: Use region-based models or add an explicit counting module.

Text in images is brittle. CLIP shows non-trivial OCR transfer, especially on rendered text, but performance drops on dense documents, small fonts, and stylized text.^[1]

• Symptom: CLIP misses label numbers on shipping boxes or misreads stylized logos.
• Cause: Web-crawled training data contains more natural-scene text than document text.
• Fix: For document OCR, use specialized document models or preprocessing pipelines with explicit OCR stages.

Bias. It inherits biases from web-crawled data, sometimes associating certain visual concepts with stereotypes.^[3]

• Symptom: Certain demographic groups or contexts get systematically lower similarity scores for neutral prompts.
• Cause: The training data reflects internet distributions, which include skewed representations.
• Fix: Audit embeddings on representative data before deployment; use debiasing techniques or domain-specific fine-tuning.

Vision encoder architectures

The vision encoder is the foundation of any VLM. Its job is to compress high-dimensional pixels into a semantic vector. This transformation must preserve essential details like shape, color, and object relationships while discarding irrelevant noise. The choice of architecture fundamentally determines the model's scalability and its ability to capture fine-grained spatial information.

Vision Transformer (ViT)^[4]

While earlier models used Convolutional Neural Networks (CNNs), like ResNet, many modern published VLMs use Vision Transformers (ViT). ViT treats images as sequences of patches, processing them with a standard Transformer encoder, much like Large Language Models (LLMs) process text tokens. The figure below illustrates this process: an input image is sliced into fixed-size patches, linearly projected into embeddings, and fed through a Transformer stack to produce a sequence of contextualized visual tokens.

Patch tokenization

A 224×224 image divided into 16×16 patches results in $14 \times 14 = 196$ visual tokens, plus a special [CLS] token used to aggregate the global image representation.

Patch size is part of the serving configuration. The example above uses 16x16 patches; the high-resolution calculation later in this chapter uses 14x14 patches. Compute the budget from the actual tower rather than copying a memorized number.

ViT vs. ResNet

SigLIP^[5]

A scaling cost in standard CLIP training is the softmax-based contrastive loss over image-text candidates in an effective batch. Original CLIP used a batch size of 32,768, which required a distributed setup; smaller implementations can still train, with a different accuracy and negative-diversity trade-off.^[1]

Instead of asking "Which of these 32,000 captions is the best match?" (a global multiple-choice question), SigLIP (Sigmoid Loss for Language-Image Pre-training) asks "Is this specific image and caption a good match, yes or no?" (a local true/false question). This improves upon CLIP by replacing the softmax-based contrastive loss with a pairwise sigmoid loss:

$$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{N} \log \sigma\big(z_{ij} \cdot (t \cdot v_i^\top u_j + b)\big)$$

Here $N$ is the batch size, $z_{ij} = +1$ for matching pairs and $z_{ij} = -1$ otherwise, while $v_i$ and $u_j$ are the L2-normalized image and text embeddings. $t$ and $b$ are learned temperature and bias parameters, and $\sigma$ is the logistic sigmoid. The original paper initializes the bias to a large negative value so that the loss starts near the all-negative prior, which stabilizes early training when most pairs in a batch are mismatches.^[5] Rather than forcing a single matching caption to compete against all others in the batch, this equation independently evaluates each pairing, treating it as a separate binary classification problem.

Advantages over CLIP

• Decoupled computation: No global normalization is needed. Each pair is scored independently, allowing the similarity matrix to be computed in chunks across devices.^[5]
• Efficiency: The sigmoid loss removes global softmax normalization. The SigLIP paper reports stronger results than its softmax baseline at studied smaller batch sizes and diminishing returns from pushing batch size much further, rather than a blanket hardware guarantee.^[5]
• Memory and partitioning: The loss is easier to chunk across devices because it doesn't require a single global softmax over the full batch.

SigLIP's pairwise sigmoid loss changes the computation contract: positive and negative pair terms can be evaluated without a batch-wide probability distribution. It does not by itself say what accelerator, data volume, or accuracy a warehouse matcher will need.

Why SigLIP matters in newer stacks

A common published VLM pattern is to swap plain CLIP-style towers for newer SigLIP-family encoders because the sigmoid objective is easier to scale and stronger at moderate batch sizes. SigLIP 2 extends that recipe with caption-based pretraining, self-distillation, masked prediction, and native-aspect-ratio support, and the paper reports better transfer than original SigLIP across model scales when used as a VLM vision backbone.^[6] The contrastive idea from CLIP is still the foundation; SigLIP changed the loss, and SigLIP 2 hardened the training recipe.

Modern VLMs: from alignment to generation

CLIP and SigLIP tell a model whether an image and a caption belong together. Modern VLMs go further: they turn visual features into something a language model can reason over token by token.

The safest way to study those systems is to stick to architectures that have actually been published. Open papers already show the main patterns we need:

• simple projectors that map vision features into the LLM space
• query-based compressors that reduce many visual patches into a fixed token budget
• resamplers and adapters that preserve more spatial detail for high-resolution inputs

That gives us a solid mental model without pretending we know the internals of proprietary systems.

LLaVA: the straightforward projector path^[7]

LLaVA is the clearest example of the "project visual features into the language model" recipe. It takes a pretrained CLIP vision encoder and a pretrained LLM, then learns a projection layer between them.

The original LLaVA recipe uses two stages:^[7]

1. Feature alignment: freeze the vision encoder and LLM, train only the projector.
2. Visual instruction tuning: fine-tune the projector and LLM on multimodal instruction data.

The architectural lesson is important: LLaVA reported useful multimodal instruction-following behavior from a pretrained vision encoder, an LLM, and a learned projector, without introducing a new vision backbone.^[7]

Common multimodal training ladder

Most strong VLMs are not trained in one jump from raw images to polished chat behavior. They are usually built in stages, with each stage solving a different problem.

CLIP and SigLIP mostly cover the first row.^[1][5] LLaVA makes the second and third rows explicit with feature alignment followed by visual instruction tuning.^[7] A chat-oriented system needs generative and instruction-following training beyond contrastive alignment, because matching alone does not teach long-form answer behavior.

This ladder is useful because it prevents a common confusion:

• contrastive pretraining teaches matching
• instruction tuning teaches how to answer
• post-training teaches which answers are preferred or allowed

If a multimodal assistant sees the right evidence but answers in the wrong format, you need better instruction tuning, not more CLIP data. If it gives polished but unsafe grounded answers, you need better post-training, not a new projector.

BLIP-2 and the Q-Former: compress before the LLM^[8]

BLIP-2 (Bootstrapping Language-Image Pre-training, version 2) is useful because it attacks a different problem. Instead of forwarding every visual feature directly into the LLM, it introduces a Q-Former that learns a fixed set of query vectors. Those queries pull the most relevant information out of the frozen vision encoder.

This gives you a bounded number of visual tokens regardless of how large the raw vision feature map is.^[8]

That is relevant in production because tokens admitted to the language path affect prefill and cache cost. If a smaller fixed budget preserves task-critical evidence, it can improve the measured latency-cost trade-off over naive concatenation.

Flamingo-style resampling: preserve more context without exploding token count^[9]

Flamingo pushed another important idea: first compress variable-length visual features with a Perceiver Resampler, then let language tokens attend to that compact bank through gated cross-attention blocks.^[9]

This kind of design is especially useful when:

• multiple images appear in one prompt
• video or long visual sequences are involved
• the language model needs to revisit visual evidence later in the answer

The core trade-off is complexity. Resamplers and cross-attention blocks preserve more structure, but they also add more moving parts than the very simple LLaVA recipe.

Qwen-VL: grounding and text reading beyond caption matching^[10]

Qwen-VL (Qwen Vision-Language) is a good published example of a chat-oriented VLM that was explicitly built for grounding and text reading, not just caption matching. The paper highlights a visual receptor, an input-output interface, and a multi-stage training pipeline that let the model handle OCR-heavy and box-grounded tasks in addition to generic image understanding.^[10]

That matters because text reading and grounding are resolution-hungry tasks.

High-resolution token math

Suppose your vision tower uses 14×14 patches. A single 336×336 crop becomes $24 \times 24 = 576$ patch tokens before any projector-side compression. Tile one document page into four such crops and you're already above 2,300 visual tokens before the model even reads the user's question.

In an autoregressive VLM, those visual tokens hurt twice: they make prefill slower, and they enlarge the KV cache that must remain resident during decode. PagedAttention helps the serving stack pack and recycle that memory more efficiently, but it doesn't reduce the number of visual tokens you created in the first place.^[11]

The connector decides how many encoder features become autoregressive prefix tokens. The example below compares raw tiles with a fixed-size compression path without claiming that compression preserves enough OCR evidence.

Production systems usually respond in one of three ways:

• resize aggressively and accept some information loss
• tile or crop the image, then summarize each region
• compress the visual stream with a Q-Former, resampler, or other projector-side bottleneck before handing it to the LLM

When a model performs well on documents, inspect the full evidence path: resolution, tiling, OCR, connector compression, and the model's measured failure slices. Published designs such as BLIP-2's Q-Former and Flamingo's resampler show ways to bound or mediate visual features before language reasoning.^[8][9]

Connector design controls a serving trade-off

The component between the vision encoder and the LLM controls what evidence reaches generation and how many visual tokens the language path must handle.

Common options include:

1. Linear projection: cheap and simple, but can bottleneck representational power.
2. MLP projector: still simple, usually stronger than a single linear layer.
3. Q-Former / query transformer: compresses many visual features into a fixed learned token set.^[8]
4. Convolutional or locality-aware abstractors: preserve more spatial structure before handing features to the LLM.^[12]

An overcompressed connector can remove evidence needed for OCR or grounding. A large or expensive connector can increase latency or language-prefix cost. Both outcomes must be measured by task slice.

That's why there's no single best adapter. The right design depends on whether you care more about:

• raw simplicity
• token efficiency
• spatial fidelity
• video or multi-image support

Designing around unpublished internals

Closed vendors may publish capability demos and benchmark numbers without enough architecture detail to justify internals claims.

Treat the model as an evaluated dependency: measure supported input resolution, visual-token billing if exposed, latency by image count, OCR and grounding slices, refusal behavior, and output contract. Do not infer a projector type, token budget, or training curriculum unless the provider publishes it.

Evaluation

Evaluating a VLM requires more than one benchmark because different tests stress very different abilities.

No single metric is enough.

• CLIP-like models can do well on retrieval and still fail badly on counting.
• Caption metrics can look fine even when spatial reasoning is weak.
• OCR-heavy tasks expose very different failure modes than general image description.

That's why serious evaluation needs a benchmark mix, not a single scoreboard.

Aggregate scores can also hide the exact slice that blocks launch. Keep capability slices separate and require task-specific pass bars.

Production Applications

Document understanding

Document AI is fundamentally a token-budget problem. Dense documents contain tiny text, tables, figures, and layout cues that don't survive aggressive resizing.

In practice, production systems usually combine some mix of:

• page tiling or crop-based preprocessing
• OCR features
• a VLM for ambiguous or layout-heavy cases

The engineering goal isn't "feed the page in at maximum resolution." It's "preserve the evidence that matters while keeping the token budget survivable."

Image search and retrieval

For image search, a generative VLM is usually the wrong first tool.

A dual-encoder model like CLIP or SigLIP is better because you can:

• precompute image embeddings offline
• index them once
• embed only the query at request time

That supports an indexed first stage whose latency and cost can be measured independently. Generative VLMs are candidates for reranking or explanation after retrieval when those added capabilities justify their measured cost.

Visual agents

Visual agents need more than a captioning model. They need grounding.

It isn't enough to answer "the blue button is near the top-right." The system often has to emit an action like:

• click (x, y)
• select the second menu item
• type into the input field beneath a label

That usually requires a grounding-aware model or a separate grounding component on top of the core VLM, plus strong safety checks around actions.

Mastery check

Use these checkpoints to test core mechanics before moving on.

Evaluation rubric

By the end of this chapter, you should be able to defend these design choices:

Follow-up questions

Common pitfalls

"One pooled CLIP embedding understands an image like a person does"

Symptom: Teams ask for counts, boxes, or grounded actions from one global embedding. Cause: CLIP was trained for global alignment, not dense spatial supervision. Fix: Use detectors, grounding-aware VLMs, or region features when localization or counting matters.

"CLIP or SigLIP can also serve my chat and captioning path"

Symptom: Similarity scores are wired into a task that needs open-ended explanations or grounded dialogue. Cause: Image-text matching was confused with multimodal generation. Fix: Keep CLIP or SigLIP for retrieval and zero-shot scoring, and use a generative VLM when you need open-ended answers.

"Higher resolution always means better OCR"

Symptom: Document quality rises a little, but latency and memory spike hard. Cause: Resolution was increased without checking token budget, tiling policy, or compression. Fix: Measure visual token count, tile selectively, preserve only the evidence that matters, and compress before the LLM when possible.

"Visual serving cost ends when the encoder finishes"

Symptom: Offline profiling looks cheap, but production decode concurrency collapses. Cause: The team counted encoder FLOPs but ignored prefill latency and KV-cache growth from large visual prefixes. Fix: Track prefill cost and KV pressure together with encoder cost before picking image resolution or crop count.

"Every in-batch non-match is a clean negative"

Symptom: Contrastive training underperforms on visually similar products or captions that describe the same item in different words. Cause: Some in-batch negatives are false negatives, so the loss pushes semantically related pairs apart. Fix: Curate data carefully, use richer captioning, and treat contrastive batch design as a modeling choice rather than a harmless training detail.

"Zero-shot prompts will hold in fine-grained domains"

Symptom: Prompt wording swings predictions or specialist categories collapse together. Cause: Generic templates do not provide enough domain signal. Fix: Prompt-ensemble, calibrate on validation data, and fine-tune or probe when domain precision matters.

What you should carry forward

After this article, you should be able to:

• Explain how CLIP's dual-encoder architecture aligns images and text in a shared embedding space.
• Walk through the contrastive loss logic with a concrete batch example.
• Implement zero-shot classification using prompt engineering and embedding similarity.
• Diagnose why CLIP fails on spatial reasoning, counting, or fine-grained OCR.
• Compare LLaVA, BLIP-2, and Flamingo-style designs for connecting vision encoders to language models.
• Size the visual token budget for a production VLM and choose an appropriate compression strategy.