Vision Transformers and Image Encoders

Text transformers start with tokens. Vision transformers start by inventing tokens from pixels.

You just learned how attention routes information across a sequence. ViT changes where the sequence comes from: image patches instead of word tokens.

That one shift explains why multimodal systems feel less mysterious. A document scan, UI screenshot, or field-inspection image can't enter a transformer as raw pixels. It must first become a sequence of vectors. A Vision Transformer (ViT) does that by cutting the image into patches, projecting each patch into an embedding, adding position information, and running transformer encoder blocks.^[1]

From pixels to patches

Take a 224 x 224 RGB image. If you use 16 x 16 patches, the image becomes:

Each patch contains 16 x 16 x 3 = 768 raw pixel values. ViT flattens each patch and applies a learned linear projection to map it to the model's hidden size d_model. ViT-Base uses d_model = 768. That this happens to equal the raw patch width of 768 is a coincidence of this configuration, not a rule: a 16 x 16 patch projected to ViT-Large's d_model = 1024 would go from 768 raw values to a 1024-dimensional vector. The raw patch width and the hidden size are two independent numbers.

More generally, for an H x W x C image and square patch width P, assuming P divides both H and W:

Track those three values separately. Changing patch size changes sequence length and raw patch width. Changing d_model changes projected feature width.

Now the transformer sees a sequence:

This is the same shape idea you learned for text. The model doesn't process "an image" directly. It processes a sequence of vectors, and attention lets each visual token look at other visual tokens.

Why flatten and project? A linear layer (a matrix multiplication) learns to turn the raw pixel values into a d_model-dimensional input representation that the transformer can refine. The weights are shared across all patches -- the same "patch embedder" is applied everywhere. This is analogous to the token embedding matrix in text transformers.

Patchify can be a strided convolution. The ViT paper describes flattening each patch and applying one shared linear projection. A 2D convolution with kernel size P and stride P can perform exactly that same operation: each output location reads one non-overlapping patch, and each output channel is one learned projection dimension. So nn.Conv2d(3, 768, kernel_size=16, stride=16) can produce the same patch tokens as explicit reshape-then-matmul when its weights and bias are copied into the equivalent linear layer.^[1]

This minimal, runnable NumPy implementation shows the patch embedding step with toy numbers:

The next check uses PyTorch's Conv2d layer and an explicit flattened-patch multiplication with identical weights. Matching output confirms that these are two implementations of the same patch projection, not two different vision architectures.

Patch size is a major design knob:

This matters when reading tiny chart labels, serial numbers, or document text. Smaller patches preserve more visual detail (a small axis label or printed ID), but self-attention cost grows with the square of token count. Systems that need fine detail often add smaller patches, higher-resolution crops, OCR fallback, or variable-resolution processing for text-heavy regions.

Position information

A bag of patches isn't enough. The model needs to know where each patch came from. A patch in the top-left corner of a inspection photo means something different from an identical patch near the barcode.

The original ViT adds a learned one-dimensional position embedding table to the raster-ordered patch sequence. A patch in row 0, column 1 receives the vector for its flattened sequence slot. The paper found no clear gain from its tested 2D-aware alternatives; the learned 1D vectors still developed row-and-column structure after training.^[1] Later encoders may use explicitly 2D or relative position schemes.

For patch sequence slot $i$, the input to the encoder is:

Original ViT also prepends a learnable classification token ([CLS]). After the encoder, that token is the image representation supplied to its classifier. Other image encoders may pool patch tokens or keep the full patch sequence for downstream tasks.

The tiny attention calculation below swaps two patch contents. A summary query can't distinguish the swap when there's no position signal; after position vectors are added, the same query produces a different summary.

The important habit is to track shape and meaning:

Learned position tables also expose a common failure when resolution changes. Moving from 224 x 224 to 256 x 256 with 16 x 16 patches changes the patch sequence from 196 to 256 entries. The original ViT approach handles higher-resolution fine-tuning by interpolating the pretrained position embeddings on their 2D patch grid.^[1]

Encoder attention

ViT uses standard transformer encoder blocks (the same scaled dot-product attention and feed-forward layers you learned in the attention chapter). Unlike a decoder-only language model, it doesn't need a causal mask for image understanding. Every patch can attend to every other patch -- the attention is bidirectional.

For a inspection image, one patch might show error panel, another might show a status badge, and another might show a tiny warning icon. Self-attention lets the encoder combine these clues across the image instead of treating each patch independently. A patch containing "ERROR" text can directly influence the representation of a distant "warning" patch.

That's why ViT connects naturally to the attention chapter. Query and key vectors still decide where information flows. Value vectors still carry content. The sequence positions are image patches instead of words. The full stack also includes residual connections, LayerNorm (covered in a later chapter), and position-wise feed-forward networks -- exactly the same building blocks as text transformers.

To see the mask difference directly, the next example uses the same attention scores twice. A vision encoder leaves all patch-to-patch entries available. A left-to-right decoder masks future entries.

The ViT architecture in full

A complete Vision Transformer typically looks like this:

1. Patch embedding (linear projection of flattened patches)
2. Prepend a learnable [CLS] token for the original classification model
3. Add learned sequence-position embeddings; some later variants encode 2D structure explicitly
4. Stack of L identical encoder blocks:
   • LayerNorm -> multi-head self-attention (bidirectional) -> residual add
   • LayerNorm -> position-wise MLP (usually 4× expansion) -> residual add
5. Final LayerNorm
6. Readout: [CLS] token, mean-pool of patch tokens, or full feature map depending on the downstream task

The output is a set of contextual visual tokens plus an optional global image representation. CLIP-style image encoders commonly read a pooled image vector, while a VLM bridge may retain many patch features.

Shape tracking example (ViT-B/16 style)

• Input image: [3, 224, 224]
• Patches: 196 × (16×16×3) = 196 × 768
• After projection: 196 × 768 tokens
• After adding position embeddings and [CLS]: 197 × 768
• After 12 encoder blocks: still 197 × 768
• Final image embedding (from CLS): 768 dimensions

This shape discipline is the same habit you developed for text token sequences.

Why pretraining scale matters. A convolutional neural network (CNN) bakes locality and translation equivariance into each layer. Original ViT uses patch extraction and learned position embeddings as its limited image-specific structure, then learns spatial relationships through attention. In the ViT paper's controlled experiments, ResNet baselines do better with smaller pretraining datasets, while ViT catches up or does better as pretraining data grows to ImageNet-21k and JFT-300M.^[1] For a limited private image dataset, begin evaluation with a pretrained encoder rather than assuming training a large ViT from scratch will win.

CLIP and image-text training

CLIP (Contrastive Language-Image Pre-training) uses contrastive learning to train an image encoder and a text encoder together so matching image-text pairs land near each other in embedding space. Its paper trains on 400 million image-text pairs gathered from the internet instead of using only a fixed class-label list.^[2]

A simplified training batch looks like this:

The model normalizes image and text embeddings, computes a matrix of scaled cosine similarities, and applies symmetric softmax cross-entropy: each image must pick its paired text, and each text must pick its paired image. For zero-shot classification, it embeds text prompts for the candidate classes and ranks their scaled similarities to the image.^[2]

CLIP isn't a captioning model by itself. It returns similarity scores. A multimodal assistant needs extra components (a language model head or a separate captioner) to turn image understanding into grounded language.

The connection to the sentence-embedding lesson is direct: the same contrastive learning machinery that turns sentences into useful vectors is used here to align two different modalities (vision and language) in one shared embedding space.

This compact inference example treats two descriptions as candidate classes. The returned number is a score-derived probability over those supplied candidates, not generated language or verified evidence.

From image encoder to VLM

Modern vision-language models (VLMs) can connect a vision encoder to a large language model (LLM). Original LLaVA uses CLIP ViT-L/14 grid features, a learned linear projection into the language embedding dimension, and a language decoder.^[3] The bridge maps each selected vision feature into an input vector the LLM can consume. Which visual features are selected, and whether they are pooled or compressed, is a model design choice rather than a ViT rule.

That handoff creates practical engineering questions about the token budget and evidence quality:

For an inspection workflow, you might ask a VLM to identify visible damage, extract an asset code, and prepare a structured request for an inspection tool. This must be evaluated separately for damage detection, text extraction, and action correctness. Visual tokens provide model input; they don't prove that a claimed detail was read correctly.

The projector itself is easy to shape-check. This scaled-down example maps vision width 8 to language-model width 16. A production bridge may use widths such as 1024 and 4096, but the invariant is identical: a linear projector changes feature width, not feature-slot count.

These counts are a patch-feature baseline, not a promise about every VLM. A classifier might use one [CLS] output. A bridge might pass all patch features, merge adjacent features, or resample them into another length.

Encoder variants change supervision and token budgets

The pipeline stays recognizable as researchers change its training objective and image sizing policy.

SigLIP: independent pair losses

CLIP's softmax loss needs normalization across rows and columns of the batch similarity matrix. SigLIP replaces this with a sigmoid loss on each image-text pair: diagonal pairs receive positive labels and off-diagonal pairs receive negative labels. The SigLIP paper reports that this loss is simpler to distribute, uses less memory in its implementation, and outperforms its softmax baseline at small batch sizes in the evaluated setups.^[4]

This code computes the pairwise logistic losses for one tiny batch. High diagonal logits and low off-diagonal logits reduce the loss.

Variable resolution and dense features

A fixed-resolution ViT setup resizes images to a chosen patch grid. Original ViT can be fine-tuned at a higher resolution by interpolating its learned position table. SigLIP 2's NaFlex variant instead supports multiple sequence lengths while preserving an image's native aspect ratio as much as its patch constraints allow.^[1][5] This is relevant to document images because squeezing a tall label into a square input changes the evidence before the model sees it.

SigLIP 2 also adds captioning-based training, self-distillation, and masked prediction; its paper reports improvements over matching SigLIP encoders on localization and dense-prediction evaluations.^[5] DINOv2 takes another route: it trains visual features without text supervision and evaluates patch-level features for segmentation and depth estimation.^[6] Which encoder to use depends on measured task behavior, not on a general rule that one backbone is best.

The mental model remains stable: patchify, project, supply position information, run encoder blocks, then choose what representation downstream code consumes. Variants change the training signal and token budget, but you still need to track shapes and preserve relevant evidence.

What to check before moving on

• Convert image size, patch width, and channels into patch count, raw patch width, and projected token shape.
• Explain why position vectors are added to raster-ordered patches and why changing resolution can require position-table interpolation.
• Trace bidirectional patch attention through a ViT encoder without introducing a causal mask.
• Distinguish CLIP similarity scoring, SigLIP pairwise training, and generative VLM behavior.
• Track how a VLM projector changes feature width while visual slots consume context-window capacity.
• Choose patch size, native resolution, cropping, or OCR based on measured detail, latency, and token-budget needs.

Practice prompts

Common failures and fixes

• Symptom: Cost estimates assume one image token equals one pixel. Cause: Patch width, channel count, and projected token count were collapsed into one idea. Fix: Track patch size, flattened patch width, patch count, and d_model separately.
• Symptom: Prototype looks fine at one resolution, then production latency jumps hard. Cause: Patch-size and image-resolution effects on visual token count were not measured. Fix: Measure emitted visual tokens on representative images before choosing encoder settings.
• Symptom: Team expects CLIP embeddings to produce grounded explanations. Cause: Retrieval-style similarity scoring was confused with generative multimodal reasoning. Fix: Use CLIP for retrieval, filtering, or zero-shot classification, and use a VLM when you need language output or tool calls.
• Symptom: Model catches broad damage but misses tiny text or torn-label details. Cause: One low-resolution encoder was asked to solve both coarse and fine tasks. Fix: Add a hybrid path with smaller patches, crops, OCR, or a detector for detail-heavy regions.
• Symptom: Large ViT trained on a small private dataset underperforms simpler baselines. Cause: In the original study, ViT benefits more from larger pretraining sets than comparable ResNet baselines. Fix: Start from a pretrained encoder, then compare it against a smaller baseline on your data.^[1]