Vision Transformers and Image Encoders

Imagine an automated monitoring bot posting a screenshot into your on-call incident channel. Across the top banner, bold red letters display "Deploy failed: Database connection timed out." Tucked into the bottom-right corner, in tiny 9-pixel gray text, sits a trace identifier: req_7f3a. Right beside it rests a blue Retry button.

Your multimodal incident responder has to accomplish two different jobs. First, it needs semantic scene understanding to recognize that the modal represents a critical deployment error. Second, it needs fine-grained character recognition to copy the exact identifier req_7f3a for an automated query. An image encoder might succeed on the error classification while completely mangling the identifier, even while a connected language model replies with grammatical confidence.

You already saw how self-attention routes information across a sequence of tokens. A Vision Transformer (ViT) uses transformer encoder blocks and changes where the sequence originates: non-overlapping image patches replace text tokens.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Our running input is a 224 x 224 pixel crop of that incident notification modal. If you treated every RGB pixel as one sequence token, that crop would produce 224 x 224 = 50,176 tokens. Full self-attention would evaluate over 2.5 billion query-key pairs per head in each layer. Patching reduces that work. A fused attention kernel can avoid storing the full score matrix, but still has to evaluate the dense pairs.

Track three quantities separately throughout this chapter: how many patches an image yields, how many raw channel values each patch holds, and how wide its projected representation becomes. The worked examples below use synthetic arrays and explicit tensors to trace these mechanics step by step.

From pixels to patches

Start with a concrete spatial calculation. If each visual token spans a square patch of 16 x 16 pixels, how many tokens come out of our 224 x 224 input crop?

Each square patch contains 16 x 16 x 3 = 768 RGB channel values. That is the patch width before the model applies any learned parameters. In a real checkpoint, follow its image processor's resize, crop, RGB conversion, and normalization rules first. Here, "raw patch width" means the number of input values, not a requirement to feed unnormalized byte intensities.

ViT flattens each patch into a one-dimensional vector $\mathbf{x}_p \in \mathbb{R}^{P^2 C}$ and multiplies it by a learned linear projection matrix $\mathbf{E} \in \mathbb{R}^{(P^2 C) \times D}$ to reach the model hidden dimension $D$ (referred to as d_model). In the standard ViT-Base architecture, $D = 768$. The matching number between 768 raw pixel values and 768 hidden units is a pure numerical coincidence: ViT-Large uses $D = 1024$, mapping 768 raw pixel numbers to a 1024-dimensional vector. Keep those two dimensions distinct in your head.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Patch width $P$ governs two separate budgets simultaneously. When you shrink patch width from 16 x 16 down to 8 x 8, the token count on that same 224 x 224 image leaps from 196 to 784. That quadruples the sequence length and multiplies dense pairwise attention comparisons by sixteen times, before counting any summary tokens.

For an arbitrary image of resolution $H \times W$ with $C$ color channels and square patch width $P$, assuming $P$ divides $H$ and $W$:

Changing $P$ alters the token count and the raw patch dimension, whereas changing d_model modifies the latent projection width. These are encoder design comparisons: an existing checkpoint's patch weights expect a particular $P^2C$ input width. Switching its patch size requires a supported adaptation method and typically further training; changing one preprocessing number is insufficient.

Once projected, the transformer receives a sequence rather than a spatial grid:

This sequence matches the shape contracts of standard language encoders. The model doesn't process the image as a 2D matrix of pixels. It operates on an ordered list of embeddings, enabling self-attention to let any visual patch exchange information with every other visual patch.

Why flatten and project? A linear layer maps raw RGB intensities into a latent representation that transformer blocks can refine. Its parameters are shared across every spatial location: the exact same patch projection runs on the top-left banner as on the bottom-right button, just like a language model uses a single token-embedding matrix across all sequence positions.

There's a clean implementation equivalent. A 2D convolution with kernel size $P$, stride $P$, no padding, and $D$ output filters reads non-overlapping patches in a single operation. Each output filter corresponds to one projection dimension. A Conv2d(3, 768, kernel_size=16, stride=16) produces the exact same numerical outputs as manual flattening followed by matrix multiplication, provided weight ordering and bias match.^{[2]Reference 2Conv2dhttps://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html}

Verify the shape transformation with a toy array. A 32 x 32 RGB image with 8 x 8 patches produces 16 patches of raw width 192, which project to width 256.

Now compare manual patch flattening against an actual PyTorch convolution layer. Notice tensor conventions: NumPy uses channels-last (HWC) arrays, while PyTorch uses channels-first (BCHW) tensors. A raw shape match won't catch transposed channel dimensions, so verify numerical equality across all tokens.

Convolutional patch projection isn't a different model; it's the exact same linear mapping executed through standard 2D convolution primitives.

Flattening complete patches is lossless: reshape them back with the same ordering to recover the input. A projection can lose information whenever its matrix lacks full input rank, even if $D$ equals or exceeds the raw patch width. A visual token is not necessarily an averaged blur. Smaller patches give attention finer spatial granularity, which can help with localized details, but do not guarantee accurate character recognition.

Look at token counts and dense query-key pairs on a fixed 224 x 224 image. These counts exclude class and register tokens:

At fixed head count and feature width, the all-to-all attention products cost $O(N^2) = O\left(\left(\frac{HW}{P^2}\right)^2\right) = O\left(\frac{H^2 W^2}{P^4}\right)$ work. The fourth-power term explains the steep increase. Materializing scores also needs quadratic space; fused attention avoids that intermediate, and windowed attention changes which pairs are evaluated.

Calculate the exact query-key pairs across settings:

Pixel-level attention generates 65,536 times more attention comparisons than 16 x 16 patching on this single image crop. Grouping pixels into patches is what makes transformer-scale attention computationally feasible on images.

Patching reduces the dense attention budget. Next, give those tokens a way to represent where each patch came from on the screen.

Position information

Two screenshots can contain the same patches arranged differently. A model that sees only an unordered bag cannot distinguish those arrangements from token order, even if patch contents offer clues about the likely scene.

Without position information or an order-dependent mask, deterministic self-attention is permutation-equivariant: shuffling tokens shuffles the output representations the same way. Mean pooling or attention with a fixed query is then invariant to that shuffle. This is the setting below, with dropout disabled. In a class-token encoder, keep the class token fixed and permute only patches: its summary remains unchanged.

Original ViT adds a learned one-dimensional position embedding table $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Using zero-based patch coordinates, row 0, column 1 is patch index 1. With the class token at sequence slot 0, that patch occupies sequence slot 2 and receives $E_{\text{pos}}[2]$.

Before passing tokens into the encoder, ViT also prepends a learnable classification token ([CLS]), designated $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$. Its output state after the final layer becomes the aggregate image representation:

The resulting input sequence $\mathbf{z}_0$ has length $N + 1$ and width $D$.

Test how adding position embeddings breaks permutation symmetry. In this experiment, two patch vectors swap locations. Without position embeddings, attention against a fixed query yields an identical summary. With position embeddings added, the two arrangements produce distinct representations:

Track the unbatched tensors for ViT-B/16 at 224-pixel resolution:

Resolution scaling and 2D position embeddings

A rigid 1D learned table creates an engineering challenge when input resolution changes. Moving from 224 x 224 to 256 x 256 with 16 x 16 patches expands sequence length from 196 to 256 patch tokens. The pretrained 1D table only has vectors for indices 0 through 196.

Original ViT uses 2D grid interpolation during higher-resolution fine-tuning.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Separate the [CLS] position vector, reshape 196 patch-position vectors into a $14 \times 14$ grid, resize to $16 \times 16$, flatten, and reattach [CLS]. The paper does not specify bicubic interpolation. Google's reference implementation uses scipy.ndimage.zoom(..., order=1), a linear interpolation method.^{[3]Reference 3Vision Transformer reference checkpoint interpolationhttps://github.com/google-research/vision_transformer/blob/main/vit_jax/checkpoint.py} Other implementations use bicubic; match the checkpoint's method and coordinate settings.

Observe the shape mismatch before any interpolation:

Try the resize itself on a rectangular grid. This synthetic example chooses PyTorch bicubic interpolation with align_corners=False, not Google's original method.^{[4]Reference 4torch.nn.functional.interpolatehttps://docs.pytorch.org/docs/2.14/generated/torch.nn.functional.interpolate.html} The three feature channels are interpolated separately, and the class position is never treated as a grid cell:

Interpolation supplies a correctly shaped table; it does not train the model to read the higher-resolution screenshot. That requires checking the checkpoint's supported resolutions and evaluating its outputs.

Some backbones, including Qwen2.5-VL, use 2D Rotary Position Embeddings (2D-RoPE).^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923[6]Reference 6RoFormer: Enhanced Transformer with Rotary Position Embeddinghttps://arxiv.org/abs/2104.09864} Rather than adding a vector to each token, they rotate queries and keys using row and column coordinates. One simple construction divides head width $d$ into two equal groups of paired dimensions: half rotate with row $y$, half with column $x$. This construction requires $d$ divisible by four; axis allocation and dimension ordering can differ between implementations.

For fixed unrotated $\mathbf{q}$ and $\mathbf{k}$, their rotated dot product contains the relative row and column offsets. The score still depends on those content vectors, too. Rotary positions avoid resizing a learned absolute table, but do not remove token limits or guarantee extrapolation quality. Qwen2.5-VL still resizes height and width to multiples of 28 and merges neighboring patches before its LLM.^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923}

Encoder attention

ViT stacks standard transformer encoder blocks: multi-head self-attention and position-wise feed-forward networks, wrapped with residual connections and LayerNorm.

Original ViT uses full bidirectional self-attention without a causal mask. Any patch can attend to any other, so the banner patch can use evidence from the Retry button in the first layer. Other vision encoders restrict visibility: Qwen2.5-VL uses local windows in most layers and full attention in four of its 32 vision layers.^{[5]Reference 5Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923} Bidirectional does not necessarily mean globally unrestricted.

Compare that bidirectional attention with a language decoder causal mask. In vision understanding, later raster patches aren't "future" steps in time; they're spatial neighbors that can provide contextual evidence immediately:

Full bidirectional attention allows distant screen regions to exchange evidence without waiting for successive local layers.

The ViT architecture in full

The end-to-end dataflow of a Vision Transformer follows a structured sequence:

ViT employs Pre-LN transformer blocks, applying LayerNorm before each sublayer and adding residuals afterward:^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

1. Patch embedding: linear projection of flattened patches from $\mathbb{R}^{P^2 C}$ to $\mathbb{R}^D$
2. Prepend a learnable [CLS] token $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$ at sequence slot 0
3. Add learned sequence position embeddings $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$
4. Pass through $L$ blocks with the same architecture and separate parameters:
   • $\mathbf{z}'_\ell = \text{MultiHeadAttention}(\text{LayerNorm}(\mathbf{z}_{\ell-1})) + \mathbf{z}_{\ell-1}$
   • $\mathbf{z}_\ell = \text{MLP}(\text{LayerNorm}(\mathbf{z}'_\ell)) + \mathbf{z}'_\ell$
5. Apply final LayerNorm
6. Readout: extract the [CLS] token for image-level tasks, or keep all $N$ patch tokens for dense vision-language handoff

For ViT-B/16, the model hidden size is $D = 768$, with $L = 12$ layers and 12 attention heads. Each head operates on dimension $d_k = 768 / 12 = 64$. Queries and keys have shape [B, 12, 197, 64], forming attention score matrices of shape [B, 12, 197, 197]. The position-wise MLP expands hidden width by $4\times$ to 3072, then projects back to 768.

Inductive biases and pretraining scale

Compare a ViT with a convolutional network such as ResNet.^{[7]Reference 7Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} Their different inductive biases influence what training has to learn.

An inductive bias represents the architectural assumptions baked directly into a model's wiring:

1. Locality: A small convolution kernel connects nearby pixels. Successive layers expand the receptive field; dilation, larger kernels, or global pooling can also connect distant regions. Downsampling is not required for receptive-field growth.
2. Translation equivariance: On an ideal grid, stride-one convolution commutes with translation: $f(T(x)) = T(f(x))$. Finite image boundaries and padding qualify this property, and strided sampling generally preserves it only for shifts aligned with the stride. It is not an exact guarantee for every shifted screenshot.

Original ViT has weaker image-specific biases than a conventional CNN, but still groups neighboring pixels into patches and uses their 2D grid when resizing position embeddings. Full attention permits long-range connections immediately; that permission does not mean the model has learned useful spatial reasoning.

In the original paper's training setup, ViTs trained on ImageNet-1K (about 1.3 million images) without strong regularization underperformed comparable ResNets. Larger-scale pretraining on ImageNet-21k or JFT improved transfer.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} This is an experimental result, not a rule that ViTs must lose on smaller datasets.

DeiT subsequently trained competitive ViTs using ImageNet alone, with a stronger training recipe and an optional distillation token.^{[8]Reference 8Training Data-Efficient Image Transformers & Distillation through Attentionhttps://arxiv.org/abs/2012.12877} ConvNeXt showed competitive convolutional backbones under updated designs and training.^{[9]Reference 9A ConvNet for the 2020shttps://arxiv.org/abs/2201.03545} Architecture, pretraining, regularization, and task all matter. For a small private screenshot set, compare pretrained encoders and smaller baselines on held-out examples before committing to a large model from scratch.

CLIP and image-text training

A trained ViT produces visual embeddings. To match our screenshot against a query like "deployment error with database timeout", we can train image and text representations in a shared space. This aligns their global readouts, rather than every image patch with every language token.

CLIP (Contrastive Language-Image Pre-training) accomplishes this using contrastive learning across a dual-encoder architecture.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

Original CLIP (2021) pairs an image encoder $f_I$ with a text encoder $f_T$ and trains on 400 million image-text pairs collected from the web.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} It evaluates both modified ResNet and ViT image encoders. Its text transformer uses a causal mask and reads the end-of-text token; its ViT image encoder uses the class token. Learned projections map those readouts into a common width $D_{\text{shared}}$, which can differ from either encoder's hidden width.

Given a training batch of $B$ paired images and captions:

1. The vision encoder and its output projection produce $B$ image vectors $u_i \in \mathbb{R}^{D_{\text{shared}}}$, then L2-normalize them: $\hat{u}_i = u_i / \|u_i\|_2$.
2. The text encoder and its output projection produce $B$ caption vectors $v_j \in \mathbb{R}^{D_{\text{shared}}}$, then L2-normalize them: $\hat{v}_j = v_j / \|v_j\|_2$.
3. Compute the $B \times B$ cosine similarity matrix scaled by a learned logit scale parameter $s = 1/\tau$: $S_{ij} = s \, (\hat{u}_i^\top \hat{v}_j)$
4. Diagonal entries ($S_{ii}$) are the supplied positive pairs. The $B^2 - B$ off-diagonal entries are assigned negative labels. Those labels are a training convention, not proof that every other caption is semantically wrong.

The training objective is the symmetric InfoNCE loss,^{[11]Reference 11Representation Learning with Contrastive Predictive Coding.https://arxiv.org/abs/1807.03748[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} composed of image-to-text and text-to-image cross-entropy terms:

$\mathcal{L}_{I \to T} = -\frac{1}{B} \sum_{i=1}^B \log \frac{\exp(S_{ii})}{\sum_{j=1}^B \exp(S_{ij})}$

$\mathcal{L}_{T \to I} = -\frac{1}{B} \sum_{j=1}^B \log \frac{\exp(S_{jj})}{\sum_{i=1}^B \exp(S_{ij})}$

$\mathcal{L}_{\text{CLIP}} = \frac{1}{2} \left(\mathcal{L}_{I \to T} + \mathcal{L}_{T \to I}\right)$

Row softmax trains each image to pick its matching caption among $B$ options; column softmax trains each caption to pick its matching image.

Suppose two screenshots both show a database timeout and both captions say "deployment failed." The diagonal-only objective treats each cross-pair as negative despite its plausible meaning. Duplicate captions cannot be distinguished by a deterministic text encoder. Batch curation and objectives supporting multiple positives can address this mismatch; increasing batch size alone does not resolve it.

Calculate the symmetric InfoNCE loss on three synthetic normalized pairs:

Zero-shot open-vocabulary classification

Because CLIP projects images and natural language into the same embedding space, it performs zero-shot open-vocabulary classification without needing a fixed, closed-set classification head.

To classify our incident screenshot across arbitrary categories:

1. Formulate prompt templates for candidate classes: "a screenshot of a deploy failed error" versus "a screenshot of a successful deployment banner".
2. Pass the candidate strings through the text encoder to obtain normalized candidate vectors $\hat{\mathbf{v}}_1, \dots, \hat{\mathbf{v}}_K$.
3. Compute the cosine similarity $\hat{\mathbf{u}}^\top \hat{\mathbf{v}}_k$ between the image embedding and each candidate text vector.
4. Select the class with the highest similarity score.

Prompt ensembling can improve zero-shot accuracy, as it did on the original paper's benchmarks.^{[10]Reference 10Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} Encode several appropriate prompt variants per class, normalize each vector, average within that class, and normalize the average again. Test the templates on your domain: a photograph template is not automatically suitable for a terminal screenshot.

The following vectors are invented for the arithmetic; the prompt strings are labels, not inputs to a real CLIP checkpoint:

Why did 0.976 become 0.494 without changing the image? Softmax distributes mass among the supplied candidates, so the duplicated prompt splits its share. These are candidate-relative probabilities, not a calibrated probability that a deployment failed. A positive common logit scale changes their sharpness without changing the similarity ranking. Neither ranking nor a sharp softmax verifies req_7f3a.

The ranking favors the deployment-failure label under the embedding model. For the next job, copying the identifier, you need a separate procedure that produces and checks a string: for example, an OCR tool or a generative VLM tested on exact IDs. To generate an explanation conditioned on the image, connect visual features to a language decoder.

From image encoder to VLM

Many generative Vision-Language Models (VLMs) connect a pretrained vision encoder to an autoregressive Large Language Model (LLM).

Original LLaVA (2023) uses a CLIP ViT-L/14 encoder and a learned linear projector.^{[12]Reference 12Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} It extracts patch-grid hidden states from a selected encoder layer, rather than CLIP's final projected global vector. For the 224-pixel CLIP checkpoint, there are $(224/14)^2 = 256$ patch features of width 1024; the class token is separate.^{[13]Reference 13CLIP ViT-L/14 checkpoint configurationhttps://huggingface.co/openai/clip-vit-large-patch14/blob/main/config.json} The LLM width depends on the chosen language checkpoint, so label it $d_{\text{LLM}}$ rather than assuming 4096.

Keep three counts separate: encoder patch states, bridge outputs, and visual slots actually inserted into LLM self-attention. The bridge determines how they relate:

A token-wise projector changes width, not count. A toy linear bridge from vision width 8 to language width 16 preserves the four input slots:

Token budget economics in VLMs

In a prefix-style VLM, each inserted visual input occupies a context window slot. Full attention over the prefix has quadratic pairwise work, while retained self-attention KV cache grows linearly with sequence length. One newly decoded token attends to the retained keys with linear lookback work. A cross-attention design such as Flamingo has separate visual memory and a different accounting boundary.

First calculate patch counts for $P=14$, before any merger:

If a prefix-style bridge passes every patch through unchanged, four $448 \times 448$ images at $P=14$ consume 4,096 visual slots, before text and any image delimiters. A 2×2 merger instead produces 256 features per image, or 1,024 total. Crops, tiling, and extra global views can increase the input count again.

Predict the storage change when one image grows from 224 to 448 pixels per side. Use a hypothetical 32-layer decoder with eight KV heads, 128-wide keys and values, two bytes per element, and 512 text tokens. Retain all prefix positions, with one patch per visual slot and no auxiliary image tokens:

Four times the visual slots only doubles the first two total prefix lengths because the 512 text slots stay fixed. The raw cache doubles too; it does not quadruple with that doubled prefix. This is payload accounting for one request, not total GPU memory: it excludes model weights, activations, allocator overhead, and cache quantization metadata. Sliding windows or other retention policies also change the cache count.

Calculate patch-feature counts for a separate $P=16$ toy encoder:

Pixel resizing, changing patch granularity, and merging encoded features are different operations. Pixel downsampling can erase character strokes before encoding; patch flattening itself does not. A learned feature merger can lose details later. Locate that boundary before trying to fix the language decoder.

Encoder variants change supervision and token budgets

Different vision backbones modify the training loss, resolution handling, or slot usage to optimize specific downstream tasks.

SigLIP: independent pair losses

CLIP normalizes each image's scores over captions and each caption's scores over images. Distributed implementations commonly gather remote embeddings to include cross-device negatives. The choice of communication collective is an implementation decision, not a mathematical requirement of softmax; a denominator can be accumulated across chunks.

SigLIP (Sigmoid Loss for Language-Image Pre-training) replaces matrix-wide softmax normalization with independent binary sigmoid losses on every pair:^{[15]Reference 15Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}

$z_{ij} = \begin{cases} +1 & \text{if } i = j \\ -1 & \text{if } i \neq j \end{cases}$

$\mathcal{L}_{ij} = \log\left(1 + \exp\left(-z_{ij} \cdot (\hat{u}_i^\top \hat{v}_j \cdot s + b)\right)\right)$

Here $s>0$ is a learned scale and $b$ is a learned bias. With $B$ diagonal positives and $B(B-1)$ negatives, the bias helps represent that imbalance. The original paper sums all pair losses and divides by $B$, not $B^2$:

$\mathcal{L}_{\text{SigLIP}} = \frac{1}{B}\sum_{i=1}^B\sum_{j=1}^B\mathcal{L}_{ij}$

Independent pair terms make chunked evaluation simpler: a chunk needs no row or column softmax denominator. SigLIP's paper exchanges negative embeddings between neighboring devices using collective permutes.^{[15]Reference 15Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343} This reduces peak memory and changes communication; it does not decouple the workers or eliminate cross-device traffic. Gradient synchronization also remains. If you discard remote negatives instead, you have changed the training pairs, not obtained the same loss for free.

The mean over four pairs is 0.28; the paper's sum divided by two examples is 0.56. Using the pair mean multiplies the objective and its gradients by $1/B$ relative to that normalization. Weighting or sampling negatives also changes the objective. Independent terms do not mean independent training decisions.

Variable resolution and dense features

Many fixed-resolution checkpoints expect a square input, but the ViT architecture itself does not require square images. Check the processor: stretching a dashboard into a square distorts characters; resizing and center-cropping can instead drop the identifier near an edge.

SigLIP 2 (2025) adds captioning and localization pretraining, followed by self-distillation and masked teacher-feature prediction, to the image-text sigmoid objective.^{[16]Reference 16SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} Its NaFlex variant largely preserves aspect ratio while resizing to patch-compatible dimensions under a chosen maximum token budget. It still interpolates a learned position grid, pads shorter sequences, and masks the padding. "Native aspect ratio" does not mean preserving every original pixel or removing context limits.

SigLIP 2 uses multi-head attention pooling for global readouts rather than assuming ViT's class-token readout.^{[16]Reference 16SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} The auxiliary captioning decoder serves training and is not included in the released encoder pair. A captioning training objective therefore does not make that checkpoint a generative captioner. Check actual auxiliary slots before adding a class token to your budget.

DINOv2 demonstrates self-supervised learning without paired text labels.^{[17]Reference 17DINOv2: Learning Robust Visual Features without Supervisionhttps://arxiv.org/abs/2304.07193} In its main pretraining setup, a student learns image-level representations and predicts teacher outputs for masked patches; the teacher sees those patches unmasked and is updated by an exponential moving average. This predicts feature distributions, not reconstructed RGB pixels. The paper evaluates segmentation and depth using readouts trained on the patch features. A backbone's feature tensor alone is not a depth map or a text-aligned classifier.

DINOv3 (released August 2025) extends this line with Gram anchoring: a teacher reference constrains relationships among patch features to counter the dense-feature degradation observed during long training.^{[18]Reference 18DINOv3https://arxiv.org/abs/2508.10104} Its family includes both ViT and ConvNeXt backbones. For our screenshot, compare exact-ID and layout-task results under the same crop and token budget; a strong result on natural-image dense prediction does not establish reliable UI transcription.

Some checkpoints introduce register tokens.^{[19]Reference 19Vision Transformers Need Registershttps://arxiv.org/abs/2309.16588} The paper observes high-norm background tokens in several tested ViTs and interprets them as slots repurposed for internal computations. Training with extra learned registers, commonly four in its experiments, removes the observed outliers and smooths feature maps in those models. This is a tested mechanism, not a universal guarantee about every ViT or every background artifact. Registers add encoder slots; whether the LLM receives them depends on the bridge's feature selection.

Trace an encoder boundary

When architecting or debugging vision-language systems:

• Calculate patch count, raw patch width, and projected tensor shapes from input resolution $H \times W \times C$ and patch width $P$.
• Account for quadratic self-attention scaling $O\left(\frac{H^2 W^2}{P^4}\right)$ before shrinking patch size to avoid memory exhaustion.
• Verify the checkpoint's position mechanism: additive tables, rotary queries and keys, or another spatial design.
• Resize learned patch-position grids with the expected interpolation settings, or use the supported rotary coordinates. Keep class and register slots separate.
• Distinguish CLIP ranking from verified transcription. Count encoder states, bridge outputs, inserted prefix slots, and retained KV separately.