Vision Transformers and Image Encoders

Imagine an automated monitoring bot posting a screenshot into your on-call incident channel. Across the top banner, bold red letters display "Deploy failed: Database connection timed out." Tucked into the bottom-right corner, in tiny 9-pixel gray text, sits a trace identifier: req_7f3a. Right beside it rests a blue Retry button.

Your multimodal incident responder has to accomplish two different jobs. First, it needs semantic scene understanding to recognize that the modal represents a critical deployment error. Second, it needs fine-grained character recognition to copy the exact identifier req_7f3a for an automated query. An image encoder might succeed on the error classification while copying req_7f3a wrong, even while a connected language model replies with grammatical confidence.

You already saw how self-attention routes information across a sequence of tokens. A Vision Transformer (ViT) uses transformer encoder blocks and changes where the sequence originates: non-overlapping image patches replace text tokens.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Our running input is a 224 x 224 pixel crop of that incident notification modal. If you treated every RGB pixel as one sequence token, that crop would produce 224 x 224 = 50,176 tokens. Full self-attention would evaluate over 2.5 billion query-key pairs per head in each layer. Patching reduces that work. A fused attention kernel can avoid storing the full score matrix, but still has to evaluate the dense pairs.

Keep three quantities separate: how many patches the crop yields, how many raw channel values each patch holds, and how wide its projected representation becomes. The worked examples use synthetic arrays so each change of shape stays visible.

View larger

From pixels to patches

Start with a concrete spatial calculation. If each visual token spans a square patch of 16 x 16 pixels, how many tokens come out of our 224 x 224 input crop?

Each square patch contains $16 \times 16 \times 3 = 768$ RGB channel values. That is the patch width before the model applies any learned parameters. In a real checkpoint, follow its image processor's resize, crop, RGB conversion, and normalization rules first. Here, "raw patch width" means the number of input values, not a requirement to feed unnormalized byte intensities.

ViT flattens each patch into a one-dimensional vector $\mathbf{x}_p \in \mathbb{R}^{P^2 C}$ and multiplies it by a learned linear projection matrix $\mathbf{E} \in \mathbb{R}^{(P^2 C) \times D}$ to reach the model hidden dimension $D$ (referred to as d_model). In the standard ViT-Base architecture, $D = 768$. The matching number between 768 raw pixel values and 768 hidden units is a pure numerical coincidence: ViT-Large uses $D = 1024$, mapping 768 raw pixel numbers to a 1024-dimensional vector. Keep those two dimensions distinct in your head.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Patch width $P$ governs two separate budgets simultaneously. When you shrink patch width from 16 x 16 down to 8 x 8, the token count on that same 224 x 224 image leaps from 196 to 784. That quadruples the sequence length and multiplies dense pairwise attention comparisons by sixteen times, before counting any summary tokens.

For an arbitrary image of resolution $H \times W$ with $C$ color channels and square patch width $P$, assuming $P$ divides $H$ and $W$:

Here, $N_{\text{patch}}$ is the token count, $D_{\text{raw}}$ is each flattened patch's input width, and $\mathbf{Z}_{\text{patch}}$ is the matrix of projected tokens. Its columns use the hidden width $D$ defined above.

Changing $P$ alters the token count and the raw patch dimension, whereas changing d_model modifies the latent projection width. These are encoder design comparisons: an existing checkpoint's patch weights expect a particular $P^2C$ input width. Switching its patch size requires a supported adaptation method and typically further training; changing one preprocessing number is insufficient.

Once projected, the transformer receives a sequence rather than a spatial grid:

[patch_1, patch_2, ..., patch_196]

This sequence matches the shape contracts of standard language encoders. The model doesn't process the image as a 2D matrix of pixels. It operates on an ordered list of embeddings, enabling self-attention to let any visual patch exchange information with every other visual patch.

Why flatten and project? A linear layer maps raw RGB intensities into a latent representation that transformer blocks can refine. Its parameters are shared across every spatial location: the exact same patch projection runs on the top-left banner as on the bottom-right button, just like a language model uses a single token-embedding matrix across all sequence positions.

There's a clean implementation equivalent. A 2D convolution with kernel size $P$, stride $P$, no padding, and $D$ output filters reads non-overlapping patches in a single operation. Each output filter corresponds to one projection dimension. A Conv2d(3, 768, kernel_size=16, stride=16) produces the exact same numerical outputs as manual flattening followed by matrix multiplication, provided weight ordering and bias match.^{[2]Reference 2Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html}

Verify the shape transformation with a toy array. A 32 x 32 RGB image with 8 x 8 patches produces 16 patches of raw width 192, which project to width 256.

import numpy as np

def extract_patches(image: np.ndarray, patch_size: int = 16) -> np.ndarray:
    """Split HxWxC image into (N_patches, patch_size*patch_size*C) flat patches."""
    if image.ndim != 3 or type(patch_size) is not int or patch_size <= 0:
        raise ValueError("expected HxWxC and a positive patch size")
    h, w, c = image.shape
    if min(h, w, c) == 0 or h % patch_size or w % patch_size:
        raise ValueError("image must contain complete, nonempty patches")
    patches = image.reshape(
        h // patch_size, patch_size,
        w // patch_size, patch_size, c
    ).swapaxes(1, 2).reshape(-1, patch_size * patch_size * c)
    return patches

def patch_embed(patches: np.ndarray, proj_weight: np.ndarray) -> np.ndarray:
    """Project (N, patch_width) patches with shared (d_model, patch_width) weights."""
    return patches @ proj_weight.T

rng = np.random.default_rng(0)
numbered = np.arange(16).reshape(4, 4, 1)
print("Raster patch rows:", extract_patches(numbered, 2).tolist())
toy_image = rng.normal(size=(32, 32, 3)).astype(np.float32)
patches = extract_patches(toy_image, patch_size=8)
print("Patches shape:", patches.shape)

d_model = 256
proj = rng.normal(size=(d_model, 192)).astype(np.float32) * 0.02
embedded = patch_embed(patches, proj)
print("Embedded tokens shape:", embedded.shape)

Raster patch rows: [[0, 1, 4, 5], [2, 3, 6, 7], [8, 9, 12, 13], [10, 11, 14, 15]]
Patches shape: (16, 192)
Embedded tokens shape: (16, 256)

Now compare manual patch flattening against an actual PyTorch convolution layer. Notice tensor conventions: NumPy uses channels-last (HWC) arrays, while PyTorch uses channels-first (BCHW) tensors. A raw shape match won't catch transposed channel dimensions, so verify numerical equality across all tokens.

import torch

torch.manual_seed(3)
image = torch.arange(2 * 3 * 4 * 6, dtype=torch.float64).reshape(2, 3, 4, 6)
patch_size = 2
d_model = 5
conv = torch.nn.Conv2d(3, d_model, patch_size, stride=patch_size).double()

n, channels, height, width = image.shape
grid_h, grid_w = height // patch_size, width // patch_size
patches = (
    image.reshape(n, channels, grid_h, patch_size, grid_w, patch_size)
    .permute(0, 2, 4, 1, 3, 5)
    .reshape(n, grid_h * grid_w, channels * patch_size * patch_size)
)
tokens = patches @ conv.weight.reshape(d_model, -1).T + conv.bias
convolved = conv(image).flatten(2).transpose(1, 2)
torch.testing.assert_close(tokens, convolved)
wrong_order = patches.reshape(n, grid_h * grid_w, channels, 4)
wrong_order = wrong_order.transpose(2, 3).reshape_as(patches)
wrong_tokens = wrong_order @ conv.weight.reshape(d_model, -1).T + conv.bias
print("Token shape:", tuple(tokens.shape))
print("All convolution tokens match:", torch.allclose(tokens, convolved))
print("Wrong channel order matches:", torch.allclose(wrong_tokens, convolved))

Token shape: (2, 6, 5)
All convolution tokens match: True
Wrong channel order matches: False

Convolutional patch projection isn't a different model; it's the exact same linear mapping executed through standard 2D convolution primitives.

Flattening complete patches is lossless: reshape them back with the same ordering to recover the input. A projection can lose information whenever its matrix lacks full input rank, even if $D$ equals or exceeds the raw patch width. A visual token is not necessarily an averaged blur. Smaller patches give attention finer spatial granularity, which can help with localized details; evaluate character accuracy on the target screenshots.

Look at token counts and dense query-key pairs on a fixed 224 x 224 image. These counts exclude class and register tokens:

View larger

At fixed head count and feature width, the all-to-all attention products cost $O(N^2) = O\left(\left(\frac{HW}{P^2}\right)^2\right) = O\left(\frac{H^2 W^2}{P^4}\right)$ work. The fourth-power term explains the steep increase. Materializing scores also needs quadratic space; fused attention avoids that intermediate, and windowed attention changes which pairs are evaluated.

Calculate the exact query-key pairs across settings:

def patch_tokens(image_size: int, patch_size: int) -> int:
    if any(type(value) is not int or value <= 0 for value in (image_size, patch_size)):
        raise ValueError("image and patch sizes must be positive integers")
    if image_size % patch_size:
        raise ValueError("image must split into complete patches")
    patches_per_side = image_size // patch_size
    return patches_per_side**2

def attention_pairs(tokens: int) -> int:
    return tokens**2

pixel_tokens = 224 * 224
tokens_16 = patch_tokens(224, 16)
tokens_14 = patch_tokens(224, 14)
tokens_8 = patch_tokens(224, 8)

print(pixel_tokens, tokens_16, tokens_14, tokens_8)
print("pixel vs P=16 pair ratio:", attention_pairs(pixel_tokens) // attention_pairs(tokens_16))
print("P=8 vs P=16 pair ratio:", round(attention_pairs(tokens_8) / attention_pairs(tokens_16), 1))
print("16-patch count correct:", tokens_16 == 196)
print("8-patch count correct:", tokens_8 == 784)

50176 196 256 784
pixel vs P=16 pair ratio: 65536
P=8 vs P=16 pair ratio: 16.0
16-patch count correct: True
8-patch count correct: True

Pixel-level attention generates 65,536 times more attention comparisons than 16 x 16 patching on this single image crop. Grouping pixels into patches is what makes transformer-scale attention computationally feasible on images.

Patching cuts the dense attention budget. The patches still need a record of where they sat, or two layouts of the same tiles can look the same.

Position information

Two screenshots can contain the same patches arranged differently. A model that sees only an unordered bag cannot distinguish those arrangements from token order, even if patch contents offer clues about the likely scene.

Without position information or an order-dependent mask, deterministic self-attention is permutation-equivariant: shuffling tokens shuffles the output representations the same way. Mean pooling or attention with a fixed query is then invariant to that shuffle. This is the setting below, with dropout disabled. In a class-token encoder, keep the class token fixed and permute only patches: its summary remains unchanged.

Original ViT adds a learned one-dimensional position embedding table $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Using zero-based patch coordinates, row 0, column 1 is patch index 1. With the class token at sequence slot 0, that patch occupies sequence slot 2 and receives $E_{\text{pos}}[2]$.

Before passing tokens into the encoder, ViT also prepends a learnable classification token ([CLS]), designated $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$. Its output state after the final layer becomes the aggregate image representation:

The resulting input sequence $\mathbf{z}_0$ has length $N + 1$ and width $D$.

Test how adding position embeddings breaks permutation symmetry. In this experiment, two patch vectors swap locations. Without position embeddings, attention against a fixed query yields an identical summary. With position embeddings added, the two arrangements produce distinct representations:

import numpy as np

def attend(query: np.ndarray, tokens: np.ndarray) -> np.ndarray:
    scores = tokens @ query / np.sqrt(tokens.shape[1])
    scores = scores - scores.max()
    weights = np.exp(scores) / np.exp(scores).sum()
    return weights @ tokens

query = np.array([1.0, 0.0])
patches = np.array([[2.0, 0.1], [0.2, 1.5]])
swapped = patches[::-1]
positions = np.array([[0.0, 0.6], [0.5, 0.0]])

without_position = np.allclose(attend(query, patches), attend(query, swapped))
with_position = np.allclose(
    attend(query, patches + positions),
    attend(query, swapped + positions),
)
print("Same summary after swap without positions:", without_position)
print("Same summary after swap with positions:", with_position)
print("Original positioned summary:", np.round(attend(query, patches + positions), 3).tolist())
print("Swapped positioned summary:", np.round(attend(query, swapped + positions), 3).tolist())

Same summary after swap without positions: True
Same summary after swap with positions: False
Original positioned summary: [1.629, 0.928]
Swapped positioned summary: [2.122, 0.429]

Track the unbatched tensors for ViT-B/16 at 224-pixel resolution:

View larger

Resolution scaling and 2D position embeddings

A rigid 1D learned table creates an engineering challenge when input resolution changes. Moving from 224 x 224 to 256 x 256 with 16 x 16 patches expands sequence length from 196 to 256 patch tokens. The pretrained 1D table only has vectors for indices 0 through 196.

Original ViT uses 2D grid interpolation during higher-resolution fine-tuning.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Separate the [CLS] position vector, reshape 196 patch-position vectors into a $14 \times 14$ grid, resize to $16 \times 16$, flatten, and reattach [CLS]. The paper does not specify bicubic interpolation. Google's reference implementation uses scipy.ndimage.zoom(..., order=1), a linear interpolation method.^{[3]Reference 3Vision Transformer reference checkpoint interpolationhttps://github.com/google-research/vision_transformer/blob/main/vit_jax/checkpoint.py} Other implementations use bicubic; match the checkpoint's method and coordinate settings.

Observe the shape mismatch before any interpolation:

import numpy as np

def sequence_length(image_size: int, patch_size: int, cls_token: bool = True) -> int:
    if any(type(value) is not int or value <= 0 for value in (image_size, patch_size)):
        raise ValueError("image and patch sizes must be positive integers")
    if type(cls_token) is not bool or image_size % patch_size:
        raise ValueError("expected a Boolean class-token flag and complete patches")
    patch_count = (image_size // patch_size) ** 2
    return patch_count + int(cls_token)

trained_positions = np.zeros((sequence_length(224, 16), 768))
higher_resolution_tokens = np.zeros((sequence_length(256, 16), 768))

print("trained slots:", trained_positions.shape[0])
print("new slots:", higher_resolution_tokens.shape[0])
print("needs resized position table:", trained_positions.shape != higher_resolution_tokens.shape)

trained slots: 197
new slots: 257
needs resized position table: True

Try the resize itself on a rectangular grid. This synthetic example chooses PyTorch bicubic interpolation with align_corners=False, not Google's original method.^{[4]Reference 4torch.nn.functional.interpolatehttps://docs.pytorch.org/docs/2.14/generated/torch.nn.functional.interpolate.html} The three feature channels are interpolated separately, and the class position is never treated as a grid cell:

import torch
import torch.nn.functional as F

def resize_patch_positions(positions, old_grid, new_grid):
    """Resize [1, 1+H*W, D] positions while preserving the class slot."""
    dimensions = (*old_grid, *new_grid)
    if len(old_grid) != 2 or len(new_grid) != 2 or any(
        type(value) is not int or value <= 0 for value in dimensions
    ):
        raise ValueError("expected two positive integer grid dimensions each")
    old_h, old_w = old_grid
    if positions.ndim != 3 or positions.shape[0] != 1 or positions.shape[-1] == 0:
        raise ValueError("expected one position table with nonempty feature width")
    if positions.shape[1] != 1 + old_h * old_w:
        raise ValueError("old grid does not match the patch positions")
    cls = positions[:, :1]
    grid = positions[:, 1:].reshape(1, old_h, old_w, -1).permute(0, 3, 1, 2)
    resized = F.interpolate(grid, size=new_grid, mode="bicubic", align_corners=False)
    patches = resized.permute(0, 2, 3, 1).reshape(1, new_grid[0] * new_grid[1], -1)
    return torch.cat((cls, patches), dim=1)

positions = torch.arange(197 * 3, dtype=torch.float64).reshape(1, 197, 3)
resized = resize_patch_positions(positions, (14, 14), (16, 20))
print("Original / resized:", tuple(positions.shape), tuple(resized.shape))
print("Class position preserved:", torch.equal(positions[:, :1], resized[:, :1]))

Original / resized: (1, 197, 3) (1, 321, 3)
Class position preserved: True

Interpolation supplies a correctly shaped table; it does not train the model to read the higher-resolution screenshot. That requires checking the checkpoint's supported resolutions and evaluating its outputs.

Some backbones, including Qwen2.5-VL, use 2D Rotary Position Embeddings (2D-RoPE).^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923[6]Reference 6RoFormer: Enhanced Transformer with Rotary Position Embeddinghttps://arxiv.org/abs/2104.09864} Rather than adding a vector to each token, they rotate queries and keys using row and column coordinates. One simple construction divides head width $d$ into two equal groups of paired dimensions: half rotate with row $y$, half with column $x$. This construction requires $d$ divisible by four; axis allocation and dimension ordering can differ between implementations.

For fixed unrotated $\mathbf{q}$ and $\mathbf{k}$, their rotated dot product contains the relative row and column offsets. The score still depends on those content vectors, too. Rotary positions avoid resizing a learned absolute table; evaluate token limits and extrapolation quality for the checkpoint. Qwen2.5-VL still resizes height and width to multiples of 28 and merges neighboring patches before its LLM.^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923}

Encoder attention

ViT stacks standard transformer encoder blocks: multi-head self-attention and position-wise feed-forward networks, wrapped with residual connections and LayerNorm.

Original ViT uses full bidirectional self-attention without a causal mask. Any patch can attend to any other, so the banner patch can use evidence from the Retry button in the first layer. Other vision encoders restrict visibility: Qwen2.5-VL uses local windows in most layers and full attention in four of its 32 vision layers.^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923} Bidirectional does not necessarily mean globally unrestricted.

Compare that bidirectional attention with a language decoder causal mask. In vision understanding, later raster patches aren't "future" steps in time; they're spatial neighbors that can provide contextual evidence immediately:

import numpy as np

def softmax(scores: np.ndarray) -> np.ndarray:
    scores = scores - scores.max(axis=-1, keepdims=True)
    probs = np.exp(scores)
    return probs / probs.sum(axis=-1, keepdims=True)

tokens = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
scores = tokens @ tokens.T / np.sqrt(tokens.shape[1])
full_weights = softmax(scores)
causal_scores = np.where(np.triu(np.ones_like(scores), k=1) == 1, -np.inf, scores)
causal_weights = softmax(causal_scores)

print("Vision patch 0 can read later patch 1:", full_weights[0, 1] > 0)
print("Causal token 0 can read later token 1:", causal_weights[0, 1] > 0)
print("Vision row sums to one:", np.isclose(full_weights[0].sum(), 1.0))

Vision patch 0 can read later patch 1: True
Causal token 0 can read later token 1: False
Vision row sums to one: True

Full bidirectional attention allows distant screen regions to exchange evidence without waiting for successive local layers.

The ViT architecture in full

A ViT builds image features in these stages:

ViT employs Pre-LN transformer blocks, applying LayerNorm before each sublayer and adding residuals afterward:^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

1. Patch embedding: linear projection of flattened patches from $\mathbb{R}^{P^2 C}$ to $\mathbb{R}^D$

2. Prepend a learnable [CLS] token $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$ at sequence slot 0

3. Add learned sequence position embeddings $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$

4. Pass through $L$ blocks with the same architecture and separate parameters. Write multi-head attention as MHA and LayerNorm as LN:

   $$\begin{aligned} \mathbf{z}'_\ell &= \operatorname{MHA}(\operatorname{LN}(\mathbf{z}_{\ell-1})) + \mathbf{z}_{\ell-1} \\ \mathbf{z}_\ell &= \operatorname{MLP}(\operatorname{LN}(\mathbf{z}'_\ell)) + \mathbf{z}'_\ell \end{aligned}$$

5. Apply final LayerNorm

6. Readout: extract the [CLS] token for image-level tasks, or keep all $N$ patch tokens for dense vision-language handoff

For ViT-B/16, the model hidden size is $D = 768$, with $L = 12$ layers and 12 attention heads. Each head operates on dimension $d_k = 768 / 12 = 64$. Queries and keys have shape [B, 12, 197, 64], forming attention score matrices of shape [B, 12, 197, 197]. The position-wise MLP expands hidden width by $4\times$ to 3072, then projects back to 768.

Inductive biases and pretraining scale

Compare a ViT with a convolutional network such as ResNet.^{[7]Reference 7Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} Their different inductive biases influence what training has to learn.

An inductive bias represents the architectural assumptions baked directly into a model's wiring:

1. Locality: A small convolution kernel connects nearby pixels. Successive layers expand the receptive field; dilation, larger kernels, or global pooling can also connect distant regions. Downsampling is not required for receptive-field growth.
2. Translation equivariance: On an ideal grid, stride-one convolution commutes with translation: $f(T(x)) = T(f(x))$. Finite image boundaries and padding qualify this property, and strided sampling generally preserves it only for shifts aligned with the stride. It is not an exact guarantee for every shifted screenshot.

Original ViT has weaker image-specific biases than a conventional CNN, but still groups neighboring pixels into patches and uses their 2D grid when resizing position embeddings. Full attention permits long-range connections immediately; evaluate whether the model has learned useful spatial reasoning.

In the original paper's training setup, ViTs trained on ImageNet-1K (about 1.3 million images) without strong regularization underperformed comparable ResNets. Larger-scale pretraining on ImageNet-21k or JFT improved transfer.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Compare models using the target dataset and training recipe to determine how they perform at smaller scale.

DeiT subsequently trained competitive ViTs using ImageNet alone, with a stronger training recipe and an optional distillation token.^{[8]Reference 8Training Data-Efficient Image Transformers & Distillation through Attentionhttps://arxiv.org/abs/2012.12877} ConvNeXt showed competitive convolutional backbones under updated designs and training.^{[9]Reference 9A ConvNet for the 2020shttps://arxiv.org/abs/2201.03545} Architecture, pretraining, regularization, and task all matter. For a small private screenshot set, compare pretrained encoders and smaller baselines on held-out examples before committing to a large model from scratch.

CLIP and image-text training

A trained ViT produces visual embeddings. To match our screenshot against a query like "deployment error with database timeout", we can train image and text representations in a shared space. This aligns their global readouts, rather than every image patch with every language token.

CLIP (Contrastive Language-Image Pre-training) accomplishes this using contrastive learning across a dual-encoder architecture.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

Original CLIP (2021) pairs an image encoder $f_I$ with a text encoder $f_T$ and trains on 400 million image-text pairs collected from the web.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} It evaluates both modified ResNet and ViT image encoders. Its text transformer uses a causal mask and reads the end-of-text token; its ViT image encoder uses the class token. Learned projections map those readouts into a common width $D_{\text{shared}}$, which can differ from either encoder's hidden width.

Given a training batch of $B$ paired images and captions:

1. The vision encoder and its output projection produce $B$ image vectors $u_i \in \mathbb{R}^{D_{\text{shared}}}$, then L2-normalize them: $\hat{u}_i = u_i / \|u_i\|_2$.

2. The text encoder and its output projection produce $B$ caption vectors $v_j \in \mathbb{R}^{D_{\text{shared}}}$, then L2-normalize them: $\hat{v}_j = v_j / \|v_j\|_2$.

3. Compute the $B \times B$ cosine similarity matrix scaled by a learned logit scale parameter $s = 1/\tau$:

   $$S_{ij} = s \, (\hat{u}_i^\top \hat{v}_j)$$

4. Diagonal entries ($S_{ii}$) are the supplied positive pairs. The $B^2 - B$ off-diagonal entries receive negative labels by the batch training convention. False negatives can occur when another caption is semantically compatible with an image.

The training objective is the symmetric InfoNCE loss,^{[11]Reference 11Representation Learning with Contrastive Predictive Coding.https://arxiv.org/abs/1807.03748[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} composed of image-to-text and text-to-image cross-entropy terms:

Row softmax trains each image to pick its matching caption among $B$ options; column softmax trains each caption to pick its matching image.

Suppose two screenshots both show a database timeout and both captions say "deployment failed." The diagonal-only objective treats each cross-pair as negative despite its plausible meaning. Duplicate captions cannot be distinguished by a deterministic text encoder. Batch curation and objectives supporting multiple positives can address this mismatch; increasing batch size alone does not resolve it.

View larger

Calculate the symmetric InfoNCE loss on three synthetic normalized pairs:

import numpy as np

images = np.eye(3)
texts = np.array([[1.0, 0.2, 0.0], [0.1, 1.0, 0.2], [0.0, 0.3, 1.0]])
texts /= np.linalg.norm(texts, axis=1, keepdims=True)
logits = 3.0 * images @ texts.T

def matching_loss(scores):
    shifted = scores - scores.max(axis=1, keepdims=True)
    log_probs = shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))
    return -np.diag(log_probs).mean()

row_loss = matching_loss(logits)
column_loss = matching_loss(logits.T)
print("Logits:", np.round(logits, 3).tolist())
print("Row / column loss:", round(row_loss, 3), round(column_loss, 3))
print("Symmetric loss:", round((row_loss + column_loss) / 2, 3))

Logits: [[2.942, 0.293, 0.0], [0.588, 2.928, 0.862], [0.0, 0.586, 2.873]]
Row / column loss: 0.155 0.156
Symmetric loss: 0.155

Zero-shot open-vocabulary classification

Because CLIP projects images and natural language into the same embedding space, it performs zero-shot open-vocabulary classification without needing a fixed, closed-set classification head.

To classify our incident screenshot across arbitrary categories:

1. Formulate prompt templates for candidate classes: "a screenshot of a deploy failed error" versus "a screenshot of a successful deployment banner".
2. Pass the candidate strings through the text encoder to obtain normalized candidate vectors $\hat{\mathbf{v}}_1, \dots, \hat{\mathbf{v}}_K$.
3. Compute the cosine similarity $\hat{\mathbf{u}}^\top \hat{\mathbf{v}}_k$ between the image embedding and each candidate text vector.
4. Select the class with the highest similarity score.

Prompt ensembling can improve zero-shot accuracy, as it did on the original paper's benchmarks.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} Encode several prompt variants suited to the domain per class, normalize each vector, average within that class, and normalize the average again. A photograph template can mismatch a terminal screenshot, so evaluate templates against the target image types.

The following vectors are invented for the arithmetic; the prompt strings are labels, not inputs to a real CLIP checkpoint:

import numpy as np

def normalize(rows: np.ndarray) -> np.ndarray:
    return rows / np.linalg.norm(rows, axis=-1, keepdims=True)

image = normalize(np.array([[0.95, 0.20, 0.05]]))
prompts = ["a deploy failed toast", "a deploy succeeded banner"]
text = normalize(np.array([[0.90, 0.25, 0.05], [0.05, 0.95, 0.10]]))
logit_scale = 5.0

logits = logit_scale * (image @ text.T)[0]
probabilities = np.exp(logits - logits.max())
probabilities = probabilities / probabilities.sum()

print("Ranked prompt:", prompts[int(np.argmax(probabilities))])
print("Candidate probabilities:", np.round(probabilities, 3).tolist())
duplicated = np.exp(np.array([logits[0], logits[0], logits[1]]) - logits.max())
duplicated /= duplicated.sum()
print("After duplicating first candidate:", np.round(duplicated, 3).tolist())

Ranked prompt: a deploy failed toast
Candidate probabilities: [0.976, 0.024]
After duplicating first candidate: [0.494, 0.494, 0.012]

Why did 0.976 become 0.494 without changing the image? Softmax distributes mass among the supplied candidates, so the duplicated prompt splits its share. These are candidate-relative probabilities, not a calibrated probability that a deployment failed. A positive common logit scale changes their sharpness without changing the similarity ranking. Neither ranking nor a sharp softmax verifies req_7f3a.

The ranking favors the deployment-failure label under the embedding model. For the next job, copying the identifier, you need a separate procedure that produces and checks a string: for example, an OCR tool or a generative VLM tested on exact IDs. To generate an explanation conditioned on the image, connect visual features to a language decoder.

From image encoder to VLM

Many generative Vision-Language Models (VLMs) connect a pretrained vision encoder to an autoregressive Large Language Model (LLM).

Original LLaVA (2023) uses a CLIP ViT-L/14 encoder and a learned linear projector.^{[12]Reference 12Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} It extracts patch-grid hidden states from a selected encoder layer, rather than CLIP's final projected global vector. For the 224-pixel CLIP checkpoint, there are $(224/14)^2 = 256$ patch features of width 1024; the class token is separate.^{[13]Reference 13CLIP ViT-L/14 checkpoint configurationhttps://huggingface.co/openai/clip-vit-large-patch14/blob/main/config.json} The LLM width depends on the chosen language checkpoint, so label it $d_{\text{LLM}}$ rather than assuming 4096.

View larger

Keep three counts separate: encoder patch states, bridge outputs, and visual slots actually inserted into LLM self-attention. The bridge determines how they relate:

A token-wise projector changes each slot's width while preserving the number of input slots. A toy linear bridge from vision width 8 to language width 16 preserves the four input slots:

import numpy as np

rng = np.random.default_rng(5)
visual_features = rng.normal(size=(1, 4, 8))
projector = rng.normal(size=(8, 16))
llm_visual_inputs = visual_features @ projector
text_token_count = 3

print("Vision features:", visual_features.shape)
print("Projected features:", llm_visual_inputs.shape)
print("Slots before generation:", llm_visual_inputs.shape[1] + text_token_count)

Vision features: (1, 4, 8)
Projected features: (1, 4, 16)
Slots before generation: 7

Token budget economics in VLMs

In a prefix-style VLM, each inserted visual input occupies a context window slot. Full attention over the prefix has quadratic pairwise work, while retained self-attention KV cache grows linearly with sequence length. One newly decoded token attends to the retained keys with linear lookback work. A cross-attention design such as Flamingo has separate visual memory and a different accounting boundary.

First calculate patch counts for $P=14$, before any merger:

If a prefix-style bridge passes every patch through unchanged, four $448 \times 448$ images at $P=14$ consume 4,096 visual slots, before text and any image delimiters. A 2×2 merger instead produces 256 features per image, or 1,024 total. Crops, tiling, and extra global views can increase the input count again.

Predict the storage change when one image grows from 224 to 448 pixels per side. Use a hypothetical 32-layer decoder with eight KV heads, 128-wide keys and values, two bytes per element, and 512 text tokens. Retain all prefix positions, with one patch per visual slot and no auxiliary image tokens:

Here, $M_{\text{KV}}$ is raw KV storage in bytes, $L$ is the number of decoder layers, $T$ is the number of retained text and visual slots, $H_{\text{KV}}$ is the number of KV heads, $d_{\text{head}}$ is each head's width, and $b_{\text{element}}$ is bytes per value. The factor of 2 counts keys and values.

layers, kv_heads, head_width, element_bytes = 32, 8, 128, 2
text_tokens = 512

def raw_kv_mib(retained_tokens):
    if type(retained_tokens) is not int or retained_tokens <= 0:
        raise ValueError("retained token count must be a positive integer")
    return 2 * layers * retained_tokens * kv_heads * head_width * element_bytes / 2**20

for image_side in (224, 448, 896):
    visual_slots = (image_side // 14) ** 2
    prefix_slots = text_tokens + visual_slots
    print(image_side, "visual / prefix:", visual_slots, prefix_slots,
          "raw KV MiB:", raw_kv_mib(prefix_slots))

print("Extra raw KV per retained token, MiB:", raw_kv_mib(1))

224 visual / prefix: 256 768 raw KV MiB: 96.0
448 visual / prefix: 1024 1536 raw KV MiB: 192.0
896 visual / prefix: 4096 4608 raw KV MiB: 576.0
Extra raw KV per retained token, MiB: 0.125

Four times the visual slots only doubles the first two total prefix lengths because the 512 text slots stay fixed. The raw cache doubles too; it does not quadruple with that doubled prefix. This is payload accounting for one request, not total GPU memory: it excludes model weights, activations, allocator overhead, and cache quantization metadata. Sliding windows or other retention policies also change the cache count.

Calculate patch-feature counts for a separate $P=16$ toy encoder:

def patch_feature_slots(image_size: int, patch_size: int, images: int = 1) -> int:
    if any(type(value) is not int or value <= 0 for value in (image_size, patch_size, images)):
        raise ValueError("sizes and image count must be positive integers")
    if image_size % patch_size:
        raise ValueError("image must split into complete patches")
    per_image = (image_size // patch_size) ** 2
    return images * per_image

base = patch_feature_slots(224, 16, images=1)
two_images = patch_feature_slots(224, 16, images=2)
high_res = patch_feature_slots(448, 16, images=1)

print(base, two_images, high_res)
print("single-image patch features correct:", base == 196)
print("two-image patch features correct:", two_images == 392)
print("high-res patch features correct:", high_res == 784)

196 392 784
single-image patch features correct: True
two-image patch features correct: True
high-res patch features correct: True

Pixel resizing, changing patch granularity, and merging encoded features are different operations. Pixel downsampling can erase character strokes before encoding; patch flattening itself does not. A learned feature merger can lose details later. Locate that boundary before trying to fix the language decoder.

Encoder variants change supervision and token budgets

The screenshot can still fail when the encoder was trained for a different target. A new loss or extra register slots changes what the model is asked to learn. Neither change automatically transcribes req_7f3a.

SigLIP: independent pair losses

CLIP normalizes each image's scores over captions and each caption's scores over images. Distributed implementations commonly gather remote embeddings to include cross-device negatives. The choice of communication collective is an implementation decision, not a mathematical requirement of softmax; a denominator can be accumulated across chunks.

SigLIP (Sigmoid Loss for Language-Image Pre-training) replaces matrix-wide softmax normalization with independent binary sigmoid losses on every pair:^{[15]Reference 15Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}

Here $s>0$ is a learned scale and $b$ is a learned bias. With $B$ diagonal positives and $B(B-1)$ negatives, the bias helps represent that imbalance. The original paper sums all pair losses and divides by $B$, not $B^2$:

Independent pair terms make chunked evaluation simpler: a chunk needs no row or column softmax denominator. SigLIP's paper exchanges negative embeddings between neighboring devices using collective permutes.^{[15]Reference 15Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343} This reduces peak memory and changes communication; it does not decouple the workers or eliminate cross-device traffic. Gradient synchronization also remains. If you discard remote negatives instead, you have changed the training pairs, not obtained the same loss for free.

import numpy as np

logits = np.array([[2.0, -1.0], [-0.4, 1.7]])
labels = 2 * np.eye(2) - 1
pair_losses = np.logaddexp(0.0, -labels * logits)

print("Positive losses:", np.round(np.diag(pair_losses), 3).tolist())
print("Negative losses:", np.round(pair_losses[labels < 0], 3).tolist())
print("Mean pair loss:", round(float(pair_losses.mean()), 3))
print("SigLIP batch normalization:", round(float(pair_losses.sum() / len(logits)), 3))

Positive losses: [0.127, 0.168]
Negative losses: [0.313, 0.513]
Mean pair loss: 0.28
SigLIP batch normalization: 0.56

The mean over four pairs is 0.28; the paper's sum divided by two examples is 0.56. Using the pair mean multiplies the objective and its gradients by $1/B$ relative to that normalization. Weighting or sampling negatives also changes the objective. Independent terms do not mean independent training decisions.

Variable resolution and dense features

Many fixed-resolution checkpoints expect a square input, but the ViT architecture itself does not require square images. Check the processor: stretching a dashboard into a square distorts characters; resizing and center-cropping can instead drop the identifier near an edge.

SigLIP 2 (2025) adds captioning and localization pretraining, followed by self-distillation and masked teacher-feature prediction, to the image-text sigmoid objective.^{[16]Reference 16SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} Its NaFlex variant largely preserves aspect ratio while resizing to patch-compatible dimensions under a chosen maximum token budget. It still interpolates a learned position grid, pads shorter sequences, and masks the padding. "Native aspect ratio" does not mean preserving every original pixel or removing context limits.

SigLIP 2 uses multi-head attention pooling for global readouts rather than assuming ViT's class-token readout.^{[16]Reference 16SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} The auxiliary captioning decoder serves training and is not included in the released encoder pair. A captioning training objective therefore does not make that checkpoint a generative captioner. Check actual auxiliary slots before adding a class token to your budget.

Padding is a correctness contract, too. A zero-valued padded key still receives positive softmax weight unless masked. Adding more such keys changes the denominator even though they contribute zero values. This synthetic fixed-query readout isolates the effect; it is not a complete SigLIP 2 implementation:

import numpy as np

def readout(tokens, valid):
    if not valid.any():
        raise ValueError("image has no valid patch tokens")
    query = np.array([1.0, 0.0])
    scores = tokens @ query / np.sqrt(tokens.shape[1])
    scores = np.where(valid, scores, -np.inf)
    weights = np.exp(scores - scores.max())
    weights /= weights.sum()
    return weights @ tokens

patches = np.array([[2.0, 0.1], [0.2, 1.5]])
padded = np.concatenate((patches, np.zeros((4, 2))))
baseline = readout(patches, np.ones(2, dtype=bool))
unmasked = readout(padded, np.ones(6, dtype=bool))
masked = readout(padded, np.array([True, True, False, False, False, False]))
assert not np.allclose(unmasked, baseline)
np.testing.assert_allclose(masked, baseline)
print("Unpadded readout:", np.round(baseline, 3).tolist())
print("Padding treated as patches:", np.round(unmasked, 3).tolist())
print("Padding masked:", np.round(masked, 3).tolist())

Unpadded readout: [1.606, 0.406]
Padding treated as patches: [0.913, 0.231]
Padding masked: [1.606, 0.406]

Repeat this check on the same processed image alone and beside a longer image in a batch, with dropout disabled. Preserve its real patch coordinates and inspect masks through every encoder layer and the readout. A masked key does not necessarily zero the output of a padded query; omit padded outputs from pooling, losses, or the LLM bridge as required by the checkpoint. Padding added after patch extraction also differs from real border pixels added before extraction. Reject an image with no valid patches instead of asking softmax to normalize an all-masked row.

DINOv2 demonstrates self-supervised learning without paired text labels.^{[17]Reference 17DINOv2: Learning Robust Visual Features without Supervisionhttps://arxiv.org/abs/2304.07193} In its main pretraining setup, a student learns image-level representations and predicts teacher outputs for masked patches; the teacher sees those patches unmasked and is updated by an exponential moving average. This predicts feature distributions, not reconstructed RGB pixels. The paper evaluates segmentation and depth using readouts trained on the patch features. A backbone's feature tensor alone is not a depth map or a text-aligned classifier.

DINOv3 (released August 2025) extends this line with Gram anchoring: a teacher reference constrains relationships among patch features to counter the dense-feature degradation observed during long training.^{[18]Reference 18DINOv3https://arxiv.org/abs/2508.10104} Its family includes both ViT and ConvNeXt backbones. For our screenshot, compare exact-ID and layout-task results under the same crop and token budget; evaluate UI transcription directly alongside natural-image dense prediction.

Some checkpoints introduce register tokens.^{[19]Reference 19Vision Transformers Need Registershttps://arxiv.org/abs/2309.16588} The paper observes high-norm background tokens in several tested ViTs and interprets them as slots repurposed for internal computations. Training with extra learned registers, commonly four in its experiments, removes the observed outliers and smooths feature maps in those models. Check for the effect in each checkpoint and artifact type you use. Registers add encoder slots; whether the LLM receives them depends on the bridge's feature selection.

Trace an encoder boundary

Follow the incident crop from pixels to the language-model prefix before you blame the decoder:

• Calculate patch count, raw patch width, and projected tensor shapes from input resolution $H \times W \times C$ and patch width $P$.
• Account for quadratic self-attention scaling $O\left(\frac{H^2 W^2}{P^4}\right)$ before shrinking patch size to avoid memory exhaustion.
• Verify the checkpoint's position mechanism: additive tables, rotary queries and keys, or another spatial design.
• Resize learned patch-position grids with the expected interpolation settings, or use the supported rotary coordinates. Keep class and register slots separate.
• Distinguish CLIP ranking from verified transcription. Count encoder states, bridge outputs, inserted prefix slots, and retained KV separately.