LearnTransformer Deep DivesVision Transformers and Image Encoders

👁️HardMultimodal Models

Vision Transformers and Image Encoders

Understand how Vision Transformers split images into patches, build visual tokens, train encoders, and connect to CLIP and multimodal LLMs.

25 min read

Learning path

Step 92 of 177 in the full curriculum

Scaled Dot-Product Attention Positional Encoding: RoPE & ALiBi

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Text transformers start with tokens. Vision transformers start by inventing tokens from pixels.

You just learned how attention routes information across a sequence. ViT changes where the sequence comes from: image patches instead of word tokens.

That one shift explains why multimodal systems feel less mysterious. A document scan, UI screenshot, or field-inspection image can't enter a transformer as raw pixels. It must first become a sequence of vectors. A Vision Transformer (ViT) does that by cutting the image into patches, projecting each patch into an embedding, adding position information, and running transformer encoder blocks.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Concrete ViT-Base shape pipeline from a 224 by 224 RGB image through 196 flattened 16 by 16 patches, shared projection to width 768, CLS and position tokens, transformer encoders, and image or patch readout — For ViT-Base, a 224 by 224 RGB image becomes 196 flattened width-768 patches. A shared projection creates 196 patch tokens, a class token expands the sequence to 197, position embeddings preserve layout, and encoder blocks return either a summary representation or contextual patch features.

From pixels to patches

Take a 224 x 224 RGB image. If you use 16 x 16 patches, the image becomes:

text

224 / 16 = 14 patches per side
14 x 14 = 196 image patches

Each patch contains 16 x 16 x 3 = 768 raw pixel values. ViT flattens each patch and applies a learned linear projection to map it to the model's hidden size d_model. ViT-Base uses d_model = 768. The fact that raw patch width and d_model are both 768 is a coincidence of this configuration, not a rule: a 16 x 16 patch projected to ViT-Large's d_model = 1024 would go from 768 raw values to a 1024-dimensional vector. The raw patch width and the hidden size are two independent numbers.

More generally, for an H x W x C image and square patch width P, assuming P divides both H and W:

text

patch_count = (H / P) x (W / P)
raw_patch_width = P x P x C
projected_tokens shape = [patch_count, d_model]

Track those three values separately. Changing patch size changes sequence length and raw patch width. Changing d_model changes projected feature width.

Now the transformer sees a sequence:

text

[patch_1, patch_2, ..., patch_196]

This is the same shape idea you learned for text. The model doesn't process "an image" directly. It processes a sequence of vectors, and attention lets each visual token look at other visual tokens.

Why flatten and project? A linear layer (a matrix multiplication) learns to turn the raw pixel values into a d_model-dimensional input representation that the transformer can refine. The weights are shared across all patches: the same "patch embedder" is applied everywhere. This is analogous to the token embedding matrix in text transformers.

Patchify can be a strided convolution. The ViT paper describes flattening each patch and applying one shared linear projection. A 2D convolution with kernel size P and stride P can perform exactly that same operation: each output location reads one non-overlapping patch, and each output channel is one learned projection dimension. So nn.Conv2d(3, 768, kernel_size=16, stride=16) can produce the same patch tokens as explicit reshape-then-matmul when its weights and bias are copied into the equivalent linear layer.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

This minimal, runnable NumPy implementation shows the patch embedding step with toy numbers:

from-pixels-to-patches.py

import numpy as np

def extract_patches(image: np.ndarray, patch_size: int = 16) -> np.ndarray:
    """Split HxWxC image into (N_patches, patch_size*patch_size*C) flat patches."""
    h, w, c = image.shape
    assert h % patch_size == 0 and w % patch_size == 0
    patches = image.reshape(
        h // patch_size, patch_size,
        w // patch_size, patch_size, c
    ).swapaxes(1, 2).reshape(-1, patch_size * patch_size * c)
    return patches

def patch_embed(patches: np.ndarray, proj_weight: np.ndarray) -> np.ndarray:
    """Project (N, patch_width) patches with shared (d_model, patch_width) weights."""
    return patches @ proj_weight.T   # shared weights for every patch

# Toy 32x32 RGB image, 8x8 patches: 16 patches, each 192 raw values
rng = np.random.default_rng(0)
toy_image = rng.normal(size=(32, 32, 3)).astype(np.float32)
patches = extract_patches(toy_image, patch_size=8)
print("Patches shape:", patches.shape)

d_model = 256
proj = rng.normal(size=(d_model, 192)).astype(np.float32) * 0.02
embedded = patch_embed(patches, proj)
print("Embedded tokens shape:", embedded.shape)

Output

Patches shape: (16, 192)
Embedded tokens shape: (16, 256)

The next check uses PyTorch's Conv2d layer and an explicit flattened-patch multiplication with identical weights. Matching output confirms that these are two implementations of the same patch projection, not two different vision architectures.

verify-convolutional-patch-embedding.py

import torch

torch.manual_seed(3)
image = torch.arange(1 * 3 * 4 * 4, dtype=torch.float32).reshape(1, 3, 4, 4)
patch_size = 2
d_model = 5

conv = torch.nn.Conv2d(3, d_model, kernel_size=patch_size, stride=patch_size)
conv_tokens = conv(image).flatten(2).transpose(1, 2)

flat_patches = (
    image.unfold(2, patch_size, patch_size)
    .unfold(3, patch_size, patch_size)
    .permute(0, 2, 3, 1, 4, 5)
    .reshape(1, -1, 3 * patch_size * patch_size)
)
linear_tokens = flat_patches @ conv.weight.reshape(d_model, -1).T + conv.bias

print("Token shape:", tuple(conv_tokens.shape))
print("Conv equals explicit projection:", torch.allclose(conv_tokens, linear_tokens))

Output

Token shape: (1, 4, 5)
Conv equals explicit projection: True

Patch size is a major design knob:

Patch size	Tokens for 224 x 224 image	Trade-off
32 x 32	49	Cheaper, less detail
16 x 16	196	Common balance
14 x 14	256	More detail, more attention cost
8 x 8	784	Expensive, useful for fine detail

Four exact patch grids for a 224 by 224 image, showing patch widths 32, 16, 14, and 8 with their patches per side, token counts, attention-pair counts, and cost relative to 16 pixel patches — For a fixed 224 by 224 image, patch width 32 produces 49 tokens and 2,401 attention pairs, while patch width 8 produces 784 tokens and 614,656 pairs. Halving patch width creates four times as many tokens and sixteen times as many query-key pairs.

This matters when reading tiny chart labels, serial numbers, or document text. Smaller patches preserve more visual detail (a small axis label or printed ID), but self-attention cost grows with the square of token count. Systems that need fine detail often add smaller patches, higher-resolution crops, OCR fallback, or variable-resolution processing for text-heavy regions.

from-pixels-to-patches-2.py

def patch_tokens(image_size: int, patch_size: int) -> int:
    assert image_size % patch_size == 0, "image must split into complete patches"
    patches_per_side = image_size // patch_size
    return patches_per_side**2

def attention_pairs(tokens: int) -> int:
    return tokens**2

tokens_16 = patch_tokens(224, 16)
tokens_14 = patch_tokens(224, 14)
tokens_8 = patch_tokens(224, 8)

print(tokens_16, tokens_14, tokens_8)
print(round(attention_pairs(tokens_8) / attention_pairs(tokens_16), 1), "x more attention pairs")
print("16-patch count correct:", tokens_16 == 196)
print("14-patch count correct:", tokens_14 == 256)
print("8-patch count correct:", tokens_8 == 784)

Output

196 256 784
16.0 x more attention pairs
16-patch count correct: True
14-patch count correct: True
8-patch count correct: True

Position information

A bag of patches isn't enough. The model needs to know where each patch came from. A patch in the top-left corner of an inspection photo means something different from an identical patch near the barcode.

The original ViT adds a learned one-dimensional position embedding table to the raster-ordered patch sequence. A patch in row 0, column 1 receives the vector for its flattened sequence slot. The paper found no clear gain from its tested 2D-aware alternatives; the learned 1D vectors still developed row-and-column structure after training.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} Later encoders may use explicitly 2D or relative position schemes.

For patch sequence slot $i$ , the input to the encoder is:

text

visual_token_i = patch_projection_i + position_embedding_i

Original ViT also prepends a learnable classification token ([CLS]). After the encoder, that token is the image representation supplied to its classifier. Other image encoders may pool patch tokens or keep the full patch sequence for downstream tasks.

Register tokens and high-norm artifacts

Modern vision encoders used as dense feature extractors or VLM backbones may add a small set of register tokens on top of patches and any [CLS]. Darcet and colleagues found high-norm artifact tokens in low-information background regions of several supervised and self-supervised ViTs. Adding learned register tokens gave those internal computations dedicated slots and produced smoother feature and attention maps in their experiments.^{[2]Reference 2Vision Transformers Need Registershttps://arxiv.org/abs/2309.16588}

Sequence-length effect: with $P$ patches, optional [CLS], and $R$ registers, the encoder sequence is about $P + [CLS] + R$ tokens. Attention cost still scales with sequence length squared, so the paper evaluates small register counts. When you only need a single global embedding and never read patch maps, registers matter less; when you ship dense features or a VLM that consumes the patch stream, check whether the backbone was trained with them and keep the same readout contract at inference.

The tiny attention calculation below swaps two patch contents. A summary query can't distinguish the swap when there's no position signal; after position vectors are added, the same query produces a different summary.

position-makes-layout-visible.py

import numpy as np

def attend(query: np.ndarray, tokens: np.ndarray) -> np.ndarray:
    scores = tokens @ query / np.sqrt(tokens.shape[1])
    scores = scores - scores.max()
    weights = np.exp(scores) / np.exp(scores).sum()
    return weights @ tokens

query = np.array([1.0, 0.0])
patches = np.array([[2.0, 0.1], [0.2, 1.5]])
swapped = patches[::-1]
positions = np.array([[0.0, 0.6], [0.5, 0.0]])

without_position = np.allclose(attend(query, patches), attend(query, swapped))
with_position = np.allclose(
    attend(query, patches + positions),
    attend(query, swapped + positions),
)
print("Same summary after swap without positions:", without_position)
print("Same summary after swap with positions:", with_position)

Output

Same summary after swap without positions: True
Same summary after swap with positions: False

The important habit is to track shape and meaning:

Tensor	Shape example	Meaning
image	`[3, 224, 224]`	RGB pixels
patches	`[196, 768]`	flattened patch values
patch embeddings	`[196, 768]`	projected visual tokens
encoder input with `[CLS]`	`[197, 768]`	summary slot plus positioned patches
encoded tokens	`[197, 768]`	contextual image representation

Vision Transformer position embedding diagram mapping a 3 by 3 raster patch grid to sequence slots, adding learned slot vectors to patch content, and showing that swapping patch contents is invisible to CLS without positions but visible with fixed slot embeddings — Raster order maps each patch to a fixed learned slot vector, so encoder input is $z_i=x_i+e_i$. Without $e_i$, permuting patch tokens leaves the class-token summary unchanged; with fixed slots, swapping two patch contents changes their content-position sums.

Learned position tables also expose a common failure when resolution changes. Moving from 224 x 224 to 256 x 256 with 16 x 16 patches changes the patch sequence from 196 to 256 entries. The original ViT approach handles higher-resolution fine-tuning by interpolating the pretrained position embeddings on their 2D patch grid.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

position-table-resolution-mismatch.py

import numpy as np

def sequence_length(image_size: int, patch_size: int, cls_token: bool = True) -> int:
    assert image_size % patch_size == 0, "image must split into complete patches"
    patch_count = (image_size // patch_size) ** 2
    return patch_count + int(cls_token)

trained_positions = np.zeros((sequence_length(224, 16), 768))
higher_resolution_tokens = np.zeros((sequence_length(256, 16), 768))

print("trained slots:", trained_positions.shape[0])
print("new slots:", higher_resolution_tokens.shape[0])
print("needs resized position table:", trained_positions.shape != higher_resolution_tokens.shape)

Output

trained slots: 197
new slots: 257
needs resized position table: True

Encoder attention

ViT uses standard transformer encoder blocks (the same scaled dot-product attention and feed-forward layers you learned in the attention chapter). Unlike a decoder-only language model, it doesn't need a causal mask for image understanding. Every patch can attend to every other patch; the attention is bidirectional.

For an inspection image, one patch might show an error panel, another might show a status badge, and another might show a tiny warning icon. Self-attention lets the encoder combine these clues across the image instead of treating each patch independently. A patch containing "ERROR" text can directly influence the representation of a distant "warning" patch.

Bidirectional Vision Transformer attention matrix for damage, label, barcode, and background patches, with every row normalized and the damage query mixing all four value vectors into one contextual token — Every query row can use every image patch. In this illustrative damage row, weights 0.38, 0.31, 0.22, and 0.09 mix damage, label, barcode, and background value vectors into one contextual representation.

That's why ViT connects naturally to the attention chapter. Query and key vectors still decide where information flows. Value vectors still carry content. The sequence positions are image patches instead of words. The full stack also includes residual connections, LayerNorm (covered in a later chapter), and position-wise feed-forward networks, the same building blocks as text transformers.

To see the mask difference directly, the next example uses the same attention scores twice. A vision encoder leaves all patch-to-patch entries available. A left-to-right decoder masks future entries.

bidirectional-image-attention.py

import numpy as np

def softmax(scores: np.ndarray) -> np.ndarray:
    scores = scores - scores.max(axis=-1, keepdims=True)
    probs = np.exp(scores)
    return probs / probs.sum(axis=-1, keepdims=True)

tokens = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
scores = tokens @ tokens.T / np.sqrt(tokens.shape[1])
full_weights = softmax(scores)
causal_scores = np.where(np.triu(np.ones_like(scores), k=1) == 1, -np.inf, scores)
causal_weights = softmax(causal_scores)

print("Vision patch 0 can read later patch 1:", full_weights[0, 1] > 0)
print("Causal token 0 can read later token 1:", causal_weights[0, 1] > 0)
print("Vision row sums to one:", np.isclose(full_weights[0].sum(), 1.0))

Output

Vision patch 0 can read later patch 1: True
Causal token 0 can read later token 1: False
Vision row sums to one: True

The ViT architecture in full

A complete Vision Transformer typically looks like this:

Patch embedding (linear projection of flattened patches)
Prepend a learnable [CLS] token for the original classification model
Add learned sequence-position embeddings; some later variants encode 2D structure explicitly
Stack of L identical encoder blocks:
- LayerNorm -> multi-head self-attention (bidirectional) -> residual add
- LayerNorm -> position-wise MLP (usually 4× expansion) -> residual add
Final LayerNorm
Readout: [CLS] token, mean-pool of patch tokens, or full feature map depending on the downstream task

The output is a set of contextual visual tokens plus an optional global image representation. CLIP-style image encoders commonly read a pooled image vector, while a VLM bridge may retain many patch features.

Shape tracking example (ViT-B/16 style)

Input image: [3, 224, 224]
Patches: 196 × (16×16×3) = 196 × 768
After projection: 196 × 768 tokens
After adding position embeddings and [CLS]: 197 × 768
After 12 encoder blocks: still 197 × 768
Final image embedding (from CLS): 768 dimensions

This shape discipline is the same habit you developed for text token sequences.

Why pretraining scale matters. A convolutional neural network (CNN) bakes locality and translation equivariance into each layer. Original ViT uses patch extraction and learned position embeddings as its limited image-specific structure, then learns spatial relationships through attention. In the ViT paper's controlled experiments, ResNet baselines do better with smaller pretraining datasets, while ViT catches up or does better as pretraining data grows to ImageNet-21k and JFT-300M.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929} For a limited private image dataset, begin evaluation with a pretrained encoder rather than assuming training a large ViT from scratch will win.

CLIP and image-text training

CLIP (Contrastive Language-Image Pre-training) uses contrastive learning to train an image encoder and a text encoder together so matching image-text pairs land near each other in embedding space. Its paper trains on 400 million image-text pairs gathered from the internet instead of using only a fixed class-label list.^{[3]Reference 3Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

A simplified training batch looks like this:

Image	Matching text
Photo of a cracked equipment panel	"cracked panel with warning label"
Instrument-panel photo	"voltage warning panel on gray rack"
Document scan	"barcode and form label"

The model normalizes image and text embeddings, computes a matrix of scaled cosine similarities, and applies symmetric softmax cross-entropy: each image must pick its paired text, and each text must pick its paired image. For zero-shot classification, it embeds text prompts for the candidate classes and ranks their scaled similarities to the image.^{[3]Reference 3Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}

CLIP batch with three normalized image vectors and three normalized text vectors forming a 3 by 3 scaled cosine logit matrix, where diagonal matches are positives, six off-diagonal cells are in-batch negatives, and row plus column cross-entropy train both retrieval directions — A batch of three matched image-text pairs creates a 3 by 3 scaled-cosine logit matrix: three diagonal positives and six off-diagonal negatives. CLIP averages image-to-text row loss and text-to-image column loss.

CLIP isn't a captioning model by itself. It returns similarity scores. A multimodal assistant needs extra components (a language model head or a separate captioner) to turn image understanding into grounded language.

The connection to the sentence-embedding lesson is direct: the same contrastive learning machinery that turns sentences into useful vectors is used here to align two different modalities (vision and language) in one shared embedding space.

This compact inference example treats two descriptions as candidate classes. The returned number is a score-derived probability over those supplied candidates, not generated language or verified evidence.

clip-style-zero-shot-ranking.py

import numpy as np

def normalize(rows: np.ndarray) -> np.ndarray:
    return rows / np.linalg.norm(rows, axis=-1, keepdims=True)

image = normalize(np.array([[0.95, 0.20, 0.05]]))
prompts = ["a cracked equipment panel", "a clean equipment panel"]
text = normalize(np.array([[0.90, 0.25, 0.05], [0.05, 0.95, 0.10]]))
logit_scale = 5.0

logits = logit_scale * (image @ text.T)[0]
probabilities = np.exp(logits - logits.max())
probabilities = probabilities / probabilities.sum()

print("Ranked prompt:", prompts[int(np.argmax(probabilities))])
print("Candidate probabilities:", np.round(probabilities, 3).tolist())

Output

Ranked prompt: a cracked equipment panel
Candidate probabilities: [0.976, 0.024]

From image encoder to VLM

Modern vision-language models (VLMs) can connect a vision encoder to a large language model (LLM). Original LLaVA uses CLIP ViT-L/14 grid features, a learned linear projection into the language embedding dimension, and a language decoder.^{[4]Reference 4Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} The bridge maps each selected vision feature into an input vector the LLM can consume. Which visual features are selected, and whether they are pooled or compressed, is a model design choice rather than a ViT rule.

Vision-language model bridge shape diagram where 196 vision features are projected from vision width to LLM width without changing slot count, concatenated with text prompt tokens, and compared with 392-slot two-image and 784-slot high-resolution budgets — A linear VLM projector changes each feature from vision width to language-model width but preserves the 196 visual slots in this baseline. Two images require 392 slots and one 448-pixel image requires 784 before any explicit pooling, resampling, or compression.

That handoff creates practical engineering questions about the token budget and evidence quality:

Concern	Engineering question
Context window	Before any compression, a 224 x 224 image at 16 x 16 patches yields 196 patch features. At 14 x 14 patches it yields 256. Does your bridge pass all features into the LLM's context window or reduce them first?
Detail preservation	Does the chosen encoder retain the label or tracking-code evidence on your evaluated images? If it doesn't, measure higher-resolution crops, optical character recognition (OCR), or a document-specific path.
Interleaving	Are visual tokens injected once at the beginning, or can the model "look again" at the image in later turns?
Tool use	Can the inspection VLM read an equipment photo, output a structured report, and then call a policy lookup tool with the detected asset tag?

For an inspection workflow, you might ask a VLM to identify visible damage, extract an asset code, and prepare a structured request for an inspection tool. This must be evaluated separately for damage detection, text extraction, and action correctness. Visual tokens provide model input; they don't prove that a claimed detail was read correctly.

The projector itself is easy to shape-check. This scaled-down example maps vision width 8 to language-model width 16. A production bridge may use widths such as 1024 and 4096, but the invariant is identical: a linear projector changes feature width, not feature-slot count.

project-visual-features-into-llm-space.py

import numpy as np

rng = np.random.default_rng(5)
visual_features = rng.normal(size=(1, 4, 8))
projector = rng.normal(size=(8, 16))
llm_visual_inputs = visual_features @ projector
text_token_count = 3

print("Vision features:", visual_features.shape)
print("Projected features:", llm_visual_inputs.shape)
print("Slots before generation:", llm_visual_inputs.shape[1] + text_token_count)

Output

Vision features: (1, 4, 8)
Projected features: (1, 4, 16)
Slots before generation: 7

from-image-encoder-to-vlm.py

def patch_feature_slots(image_size: int, patch_size: int, images: int = 1) -> int:
    assert image_size % patch_size == 0, "image must split into complete patches"
    per_image = (image_size // patch_size) ** 2
    return images * per_image

base = patch_feature_slots(224, 16, images=1)
two_images = patch_feature_slots(224, 16, images=2)
high_res = patch_feature_slots(448, 16, images=1)

print(base, two_images, high_res)
print("single-image patch features correct:", base == 196)
print("two-image patch features correct:", two_images == 392)
print("high-res patch features correct:", high_res == 784)

Output

196 392 784
single-image patch features correct: True
two-image patch features correct: True
high-res patch features correct: True

These counts are a patch-feature baseline, not a promise about every VLM. A classifier might use one [CLS] output. A bridge might pass all patch features, merge adjacent features, or resample them into another length.

Encoder variants change supervision and token budgets

The pipeline stays recognizable as researchers change its training objective and image sizing policy.

SigLIP: independent pair losses

CLIP's softmax loss needs normalization across rows and columns of the batch similarity matrix. SigLIP replaces this with a sigmoid loss on each image-text pair: diagonal pairs receive positive labels and off-diagonal pairs receive negative labels. The SigLIP paper reports that this loss is simpler to distribute, uses less memory in its implementation, and outperforms its softmax baseline at small batch sizes in the evaluated setups.^{[5]Reference 5Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}

This code computes the pairwise logistic losses for one tiny batch. High diagonal logits and low off-diagonal logits reduce the loss.

siglip-pairwise-loss.py

import numpy as np

logits = np.array([[2.0, -1.0], [-0.4, 1.7]])
labels = 2 * np.eye(2) - 1
pair_losses = np.logaddexp(0.0, -labels * logits)

print("Positive losses:", np.round(np.diag(pair_losses), 3).tolist())
print("Negative losses:", np.round(pair_losses[labels < 0], 3).tolist())
print("Mean pair loss:", round(float(pair_losses.mean()), 3))

Output

Positive losses: [0.127, 0.168]
Negative losses: [0.313, 0.513]
Mean pair loss: 0.28

Variable resolution and dense features

A fixed-resolution ViT setup resizes images to a chosen patch grid. Original ViT can be fine-tuned at a higher resolution by interpolating its learned position table. SigLIP 2's NaFlex variant instead supports multiple sequence lengths while preserving an image's native aspect ratio as much as its patch constraints allow.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}^{[6]Reference 6SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} This is relevant to document images because squeezing a tall label into a square input changes the evidence before the model sees it.

SigLIP 2 also adds captioning-based training, self-distillation, and masked prediction; its paper reports improvements over matching SigLIP encoders on localization and dense-prediction evaluations.^{[6]Reference 6SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featureshttps://arxiv.org/abs/2502.14786} DINOv2 takes another route: it trains visual features without text supervision and evaluates patch-level features for segmentation and depth estimation.^{[7]Reference 7DINOv2: Learning Robust Visual Features without Supervisionhttps://arxiv.org/abs/2304.07193} Which encoder to use depends on measured task behavior, not on a general rule that one backbone is best.

The mental model remains stable: patchify, project, supply position information, run encoder blocks, then choose what representation downstream code consumes. Variants change the training signal and token budget, but you still need to track shapes and preserve relevant evidence.

What to check before moving on

Convert image size, patch width, and channels into patch count, raw patch width, and projected token shape.
Explain why position vectors are added to raster-ordered patches and why changing resolution can require position-table interpolation.
Trace bidirectional patch attention through a ViT encoder without introducing a causal mask.
Distinguish CLIP similarity scoring, SigLIP pairwise training, and generative VLM behavior.
Track how a VLM projector changes feature width while visual slots consume context-window capacity.
Choose patch size, native resolution, cropping, or OCR based on measured detail, latency, and token-budget needs.

Practice prompts

Common failures and fixes

Symptom: Cost estimates assume one image token equals one pixel. Cause: Patch width, channel count, and projected token count were collapsed into one idea. Fix: Track patch size, flattened patch width, patch count, and d_model separately.
Symptom: Prototype looks fine at one resolution, then production latency jumps hard. Cause: Patch-size and image-resolution effects on visual token count were not measured. Fix: Measure emitted visual tokens on representative images before choosing encoder settings.
Symptom: Team expects CLIP embeddings to produce grounded explanations. Cause: Retrieval-style similarity scoring was confused with generative multimodal reasoning. Fix: Use CLIP for retrieval, filtering, or zero-shot classification, and use a VLM when you need language output or tool calls.
Symptom: Model catches broad damage but misses tiny text or torn-label details. Cause: One low-resolution encoder was asked to solve both coarse and fine tasks. Fix: Add a hybrid path with smaller patches, crops, OCR, or a detector for detail-heavy regions.
Symptom: Large ViT trained on a small private dataset underperforms simpler baselines. Cause: In the original study, ViT benefits more from larger pretraining sets than comparable ResNet baselines. Fix: Start from a pretrained encoder, then compare it against a smaller baseline on your data.^{[1]Reference 1An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.https://arxiv.org/abs/2010.11929}

Next Step

Continue to Positional Encoding: RoPE & ALiBi

ViT showed why position matters for image patches; <span data-glossary="positional-encoding">positional encoding</span> now shows how transformer systems represent order and distance for text and long-context sequence modeling.

PreviousScaled Dot-Product Attention

Share this article