Multimodal LLM Architecture

Vision-language models (VLMs) and CLIP (Contrastive Language-Image Pre-training)^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} showed how image-text alignment and visual token budgets work. Multimodal large language model (LLM) architecture generalizes that into a system stack: modality encoders, connector design, fusion strategy, training recipe, serving cache, and safety boundary.

This design chapter explains how encoders, connectors, shared representations, and operational constraints align text, images, audio, and other signals into one reasoning system.

An on-call assistant may need to inspect a dashboard screenshot, read the attached incident transcript, and use both to decide whether the issue is a deploy, quota, or service dependency problem. Text, images, and audio arrive in different forms: a pixel has no obvious relationship to a word.

For a model to answer from those inputs, it needs a trained interface between modality features and its language path. This is the core of multimodal large language model (LLM) design: deciding what visual or audio evidence is encoded, what is compressed, and how the decoder can use it.

Product brief: an incident-evidence copilot

Design a copilot that accepts a dashboard screenshot, a call transcript or audio clip, and an operator question. It returns a diagnosis only when it can cite the text span, chart region, or tool result that supports the answer.

Requirements and API

• POST /v1/cases accepts immutable image, audio, and transcript references plus tenant, incident, and idempotency keys. It returns case_id and per-modality processing state.
• POST /v1/cases/{case_id}/questions accepts question and a latency class. It returns answer, status (grounded, partial, abstained, or review), evidence IDs with text spans or image regions, and encoder, connector, model, and policy versions.
• Interactive questions target p95 below 3 seconds for one image and an existing transcript. Audio transcription and long-video processing may finish asynchronously.
• The system must isolate tenants, treat image-derived instructions as untrusted evidence, and refuse claims whose cited region or transcript span is missing.

Data flow and sizing

The path is artifact admission -> format and policy checks -> vision/audio/text frontends -> versioned feature store -> connector or reducer -> fusion and language model -> evidence validator -> policy gate -> answer trace. Raw artifacts remain immutable, while each derived feature records its source hash and model version.

Suppose one dashboard uses a global thumbnail plus four crops. A direct ViT path at 576 patch tokens per view would create 5 x 576 = 2,880 visual tokens. A tested reducer that emits 32 tokens per view lowers that to 160 before adding a 2,000-token transcript and 300-token output reservation. At 20 question QPS, use separate encoder and decoder pools so bursty image work, cached-case follow-ups, and audio backfills can scale independently.

Recovery, rollout, and evaluation

Each modality stage is idempotent and records pending, ready, failed, or quarantined. A failed image encoder can retry without rerunning completed transcription. If one modality is unavailable, the API may return a clearly labeled partial answer only when product policy permits it; otherwise it abstains. Never silently substitute language priors for missing visual evidence.

Roll out one modality and connector version at a time: frozen offline cases, shadow comparison, a tenant-limited canary, then staged expansion with the old feature generation available for rollback. Gate on answer faithfulness, OCR exact match, region-citation IoU, transcript word error rate, abstention quality, prompt-injection tests, p95 latency, GPU memory, and cost per grounded answer. Slice results by modality combination, evidence length, image resolution, audio quality, tenant, and failure path.

The concept: one shared case file

A multimodal model behaves like an operations desk that merges signals from several systems into one case file.

• The Text Path turns incident messages, runbook snippets, and trace summaries into token embeddings (using a tokenizer) that the language model can reason over.
• The Vision Encoder turns dashboard screenshots, hardware-label photos, or whiteboard diagrams into dense visual features.
• The Connector maps or exposes those visual features to the language path, through projected prefix tokens, cross-attention memory, or another published interface.

Another way to picture this is evidence routing. The vision encoder turns image evidence into feature records. The language model already works with text token states. A connector makes image features consumable by the language path; it doesn't automatically prove that "axis label" pixels and words are aligned or that the answer is grounded. Alignment and grounding must be trained and evaluated.

Architecture components

Adapter-based vs jointly trained stacks

Published multimodal systems include two useful patterns. Adapter-based models keep a strong modality encoder and text LLM, then learn a connector between them. BLIP-2 (Bootstrapping Language-Image Pre-training, version 2)^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} and LLaVA (Large Language-and-Vision Assistant)^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} are examples. Jointly trained multimodal stacks, such as PaLI (Pathways Language and Image)^{[4]Reference 4PaLI: A Jointly-Scaled Multilingual Language-Image Model.https://arxiv.org/abs/2209.06794} and Gemini^{[5]Reference 5Gemini: A Family of Highly Capable Multimodal Models.https://arxiv.org/abs/2312.11805}, optimize more of the multimodal stack together on multimodal mixtures.

For concrete published or disclosed examples: Llama 4 describes early fusion over interleaved text, image, and video data^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/}; OpenAI describes GPT-4o as trained end-to-end across text, vision, and audio^{[7]Reference 7GPT-4o System Card.https://arxiv.org/abs/2410.21276}; Qwen2.5-VL describes a dynamic-resolution vision encoder trained with its language stack.^{[8]Reference 8Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923} These reports establish architecture families, not a universal ranking over adapter-based and joint approaches.

Adapter-based designs let a team reuse pretrained components and train fewer parameters. Joint training exposes more parameters and modalities to optimization, raising data, compute, and stability requirements. Choose from measured target-task quality, budget, and controllability rather than assuming one family wins.

Modality-specific encoders

A practical design starts by asking how each non-text modality becomes task-useful features. In encoder-connector stacks, modality frontends compress raw signals (such as noisy RGB pixels or high-frequency audio waveforms) into dense mathematical representations.

Different modality inputs flow through separate encoders:

Key observations

• Pre-trained vs. Scratch: Pre-trained encoders (e.g., CLIP^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020}, SigLIP^{[9]Reference 9Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343}) are practical starting points when their visual domain overlaps your task.
• Frozen vs. Fine-tuned: Freezing the encoder initially is one stable alignment recipe; later unfreezing is an experiment with quality and forgetting risks.
• Backbone Selection: CLIP ViT and SigLIP are standard starting points for image encoders. For audio, Whisper^{[10]Reference 10Whisper: Robust Speech Recognition via Large-Scale Weak Supervision.https://arxiv.org/abs/2212.04356} is common for speech-heavy pipelines, while CLAP (Contrastive Language-Audio Pretraining)^{[11]Reference 11CLAP: Learning Audio Concepts from Natural Language Supervision.https://arxiv.org/abs/2206.04769} is better for general audio-text alignment than transcription.

Connectors and projection layers

Non-text modalities produce feature tensors that don't directly match the decoder interface. In a projected-prefix design, a projector maps features into token-shaped states that enter the decoder context. In a cross-attention design, a connector can instead expose a separate visual memory bank.

The connector determines what evidence the decoder receives. If this step throws away spatial detail or its alignment training fails, the decoder may sound fluent while grounding poorly in the input.

A connector can be as simple as a linear projector or as structured as a learned bottleneck with cross-attention. The runnable example below models the systems consequence: some connectors keep every visual feature, while Q-Former and Perceiver-style reducers bound the downstream visual stream before a prefix or cross-attention path exposes it.

Connector methods compared

Q-Former and Perceiver-style resamplers address the same systems problem: bound visual features before the language path, with potential information loss to evaluate. BLIP-2^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} uses a Q-Former. Flamingo^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} uses a Perceiver Resampler to produce a fixed visual bank before gated cross-attention layers.

The token count problem

A 224×224 image processed by a ViT-L/14 (Vision Transformer Large with a patch size of 14) encoder produces a 16×16 patch grid, or 256 patch tokens. Some implementations also keep a CLS token, yielding 257 total encoder tokens. Raise the resolution and the count climbs fast: LLaVA-1.5 swapped in a 336px CLIP encoder, which yields a 24×24 grid and 576 visual tokens per image.^{[13]Reference 13Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744} In a direct projected-prefix path, admitting thousands of visual tokens increases shared self-attention work, prefill length, and KV-cache demand. A separate cross-attention bank has a different cost path.

Worked example: how many tokens does one image cost?

The arithmetic is short enough to do in an interview or on a whiteboard.

• Input: a single 224×224 image.
• Encoder: ViT-L/14, which means the patch size is 14 pixels.
• Step 1: Divide the image width by the patch size: $224 \div 14 = 16$ patches across.
• Step 2: The height gives the same count, so the total patches are $16 \times 16 = 256$.
• Result: The image becomes 256 visual tokens before compression. In a direct projected-prefix route with a 4096-token context window, one image alone eats roughly 6% of your total budget. Add a second image and you've spent 12%. Add a one-minute video at two frames per second and you're looking at 30,720 raw patch tokens, or 7.5 times that route's full allowance, before temporal compression.

Visual token budgeting is a first-class design decision in architectures that admit visual tokens to the decoder or repeatedly attend over visual memory.

Solutions

1. Spatial compression: Pooling or resampling visual tokens (e.g., 256 → 64).
2. Temporal compression: Sampling keyframes or using temporal attention layers (e.g., TimeSformer (Time-Space Transformer)^{[14]Reference 14Is Space-Time Attention All You Need for Video Understanding?https://arxiv.org/abs/2102.05095}, VideoMAE (Video Masked Autoencoders)^{[15]Reference 15VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training.https://arxiv.org/abs/2203.12602}).
3. Dynamic resolution: Tiling or cropping high-resolution inputs so detailed regions get more compute than the rest of the image.

Fusion strategies

Compare where text and visual features meet. Early fusion mixes tokens into one sequence before the transformer. Cross-attention lets configured language layers read visual memory. Late fusion pools each side independently and only merges at the end. None of these placements guarantees grounding; each defines what evidence and serving cost the model can incur.

Early fusion (visual prefix or interleaving)

In early fusion, projected modality tokens are inserted directly into the model input sequence, usually as a contiguous visual prefix or at special placeholder positions inside the prompt. After the projector creates decoder-compatible token states, the model treats image tokens and text tokens as one long sequence and runs standard self-attention over the whole thing. This is used in LLaVA-style architectures and published native multimodal transformers such as Llama 4:

Pros

Provides token-level image-text interaction through shared self-attention from the first decoder layer that receives the prefix.

Cons

Computationally expensive for long visual sequences, as image tokens consume a large portion of the shared context-window capacity.

Late fusion

Late fusion is common in dual-encoder retrieval or classification systems. Each modality is encoded mostly independently, pooled into a compact embedding, and merged by a lightweight head. This design supports indexed scoring, but can't by itself produce answers grounded in detailed visual tokens because a generative decoder is absent from that path:

Pros

Highly modular for scoring. Encoders can be swapped independently, and a retrieval path avoids a long generative multimodal prefix.

Cons

Insufficient by itself for grounded generation. Because the scoring path never presents token-level image evidence to a decoder, optical character recognition (OCR)-heavy answering or region-cited reasoning needs an additional model path.

Cross-attention fusion (Flamingo-style)

Cross-attention layers allow text states to compute context-dependent weights over visual features at configured model depths, as seen in Flamingo.^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} Softmax turns the compatibility scores into weights. The gated layer takes the LLM's hidden text state and frozen visual features as inputs, returning a fused representation. Flamingo initializes the gate at zero so training begins from the language-only path and can learn to admit visual information:

$$\text{Attention}(Q_{\text{text}}, K_{\text{vis}}, V_{\text{vis}}) = \text{softmax}\left(\frac{Q_{\text{text}} K_{\text{vis}}^T}{\sqrt{d_k}}\right) V_{\text{vis}}$$

Architecture

Gated cross-attention is injected between standard LLM layers to incorporate frozen visual features:

At initialization, a zero gate leaves the language path unchanged by the visual residual. Training can then learn non-zero visual contributions.

Compared with early fusion, this keeps dense visual tokens out of the main decoder prefix. It doesn't make vision free: every configured cross-attention block still performs extra projections and attention against visual memory.

Training strategies

Two-stage training (LLaVA protocol)

The reported LLaVA recipe for visual instruction tuning^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}, later refined in LLaVA-1.5^{[13]Reference 13Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744}:

Stage 1: Feature Alignment

• Freeze: Vision encoder + LLM
• Train: Projection layer only
• Data: ~595K image-text pairs from a filtered CC3M (Conceptual Captions 3M) subset^{[16]Reference 16Conceptual Captions: A Cleaned, Hyperponymed, Image-Caption Dataset for Automatic Image Captioning.https://aclanthology.org/N18-2012/}
• Objective: Predict caption tokens autoregressively through the frozen LLM while training only the projection layer.

Stage 2: Visual Instruction Tuning

• Freeze: Vision encoder (in the reported LLaVA recipe)
• Train: Projection layer + LLM (full fine-tune in the reported recipe; LoRA is a lower-cost implementation choice)
• Data: ~158K GPT-4-generated multimodal instructions across conversations, detailed descriptions, and complex reasoning^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}.
• Objective: Learn to follow instructions and reason about visual content.

Those counts describe original LLaVA recipe: a filtered CC3M pretraining set plus 158K GPT-4-generated multimodal instructions^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}. LLaVA-1.5 kept same two-stage shape, but swapped in a 336px CLIP encoder, an MLP connector, and a larger public task mixture. Its paper reports roughly 1.2M total training examples and full training in about one day on a single 8-A100 node^{[13]Reference 13Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744}.

For custom domains like dashboard inspection, UI screenshot QA, or complex charts, compare a connector-alignment stage against direct instruction tuning on held-out grounding slices. Stage ordering is an experimental choice outside the reported LLaVA recipe, not a guaranteed improvement.

The two stages connect like this:

End-to-end training

While two-stage training keeps the pre-trained components largely frozen, end-to-end training updates the modality encoder, projector, and core transformer together. Jointly trained families such as PaLI^{[4]Reference 4PaLI: A Jointly-Scaled Multilingual Language-Image Model.https://arxiv.org/abs/2209.06794}, Gemini^{[5]Reference 5Gemini: A Family of Highly Capable Multimodal Models.https://arxiv.org/abs/2312.11805}, and Llama 4^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/} represent this direction and aim for a more native multimodal representation. Llama 4 warm-starts from a MetaCLIP-based vision encoder that was first aligned against a frozen Llama, then trains the whole stack on interleaved data; in this published recipe, joint training doesn't mean starting every component from random weights^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/}.

This strategy allows gradients to update cross-modal representations jointly. Instead of holding the vision encoder fixed, training can adapt its features to downstream objectives. Whether this improves chart reading, OCR, or grounding depends on data, optimization, and evaluation.

However, joint training exposes more parameters and objectives to optimization. It requires substantial compute and balanced datasets containing text and multimodal examples. Multimodal updates can conflict with language-modeling behavior, so text-only regression measurement remains part of the training contract.

Trade-offs

• Compute: Higher training cost than a projector-only stage because gradients update the encoder and language model as well.
• Stability: Harder to stabilize; carefully tuned learning rates and loss weighting are required to prevent one modality from dominating.
• Quality: Can adapt the encoder to downstream objectives; measure gains against adapter baselines on target slices.

Post-training for grounded behavior

After instruction tuning, a system stack may add a post-training stage focused on response quality, abstention, and safety. Direct Preference Optimization (DPO)^{[17]Reference 17Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} is one option, but it's not a universal third stage for multimodal systems. Depending on the route, teams may use supervised preference data, rejection sampling, RLHF, or task-specific evaluation loops instead.

The important systems point is that post-training can improve how the model explains uncertainty, refuses unsupported claims, and follows system policy. It can't recover information that was already lost upstream. If the vision encoder missed small text or the projector over-compressed the image, no preference method can reconstruct that detail later.

If a multimodal model hallucinates image details, inspect encoder resolution, connector bottleneck, alignment data, and post-training separately. If required evidence never reaches the decoder, preference tuning can't reconstruct it.

Efficient inference: token budget before parameter budget

Multimodal latency includes modality encoding and processing visual features inside the decoder. In early-fusion systems, that second cost can show up as a large prefill over visual tokens. In cross-attention systems, the prefix stays smaller, but each fusion block still attends over visual memory. Measure these components by route before choosing whether to compress inputs or optimize decode. PagedAttention-style serving makes large, irregular KV caches easier to manage under batching pressure^{[18]Reference 18Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}.

If users ask multiple questions about the same image, caching encoder features avoids rerunning the vision frontend. Caching projected tokens avoids repeating projection. Only a compatible prefix/KV-cache reuse mechanism avoids repeating decoder prefill over an identical visual prefix.

Where MoE helps, and where it doesn't

Mixture-of-Experts (MoE) layers activate only a subset of experts per token, which lowers active FLOPs at a given parameter count^{[19]Reference 19Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961}. That can improve throughput for very large multimodal backbones, but it doesn't remove the main multimodal bottlenecks:

• Vision or audio encoders still have to run.
• In early-fusion systems, long visual prefixes still bloat the KV cache.
• High-resolution inputs still create expensive prefills.

Treat MoE as a model-capacity tool, not as the first fix for high-resolution image serving.

Handling variable-length visual tokens

A major bottleneck in multimodal architectures is managing the explosion of tokens when processing high-resolution images or long videos. Keeping every patch token from every crop or frame can be computationally prohibitive. In early-fusion paths, self-attention grows quadratically with the shared sequence length. Cross-attention paths avoid that prefix growth, but still pay to read the visual bank. Two prominent strategies control this budget while preserving important details.

Tiling and multi-crop encoding

One common way to preserve high-resolution detail is to encode a global thumbnail plus a small set of crops chosen by a layout, OCR, or task policy. The thumbnail preserves overall layout, while selected crops can preserve fine text, tables, or localized objects. Crop selection is itself an evaluated component: a skipped region can't be recovered downstream.

More crops improve OCR and small-object reasoning, but every crop adds another encoder pass and another batch of visual tokens. That makes this strategy best for documents, charts, maps, and other inputs where fine detail matters.

Newer native multimodal models push this further. Qwen2.5-VL processes images at their native resolution with a dynamic-resolution ViT, then merges each 2×2 patch block into one token so the visual token count tracks actual image size instead of a fixed grid^{[8]Reference 8Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923}. The systems goal is the same as tiling: spend visual tokens where detail lives, and keep the budget bounded everywhere else.

Because the number of crops depends on aspect ratio and resolution, the resulting token sequence length is variable. A wide panorama might need a horizontal strip of crops, while a long document might need a vertical stack:

Queried reduction (Perceiver / Q-Former^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597})

This approach (popularized by models like Flamingo^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} and BLIP-2^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597}) uses a fixed set of learned query vectors to scan the visual features and compress them into a predefined token count. By decoupling the visual input resolution from the sequence length passed to the language model, this architecture helps prevent out-of-memory failures and keeps the effective visual token budget bounded even for high-framerate video or dense spatial layouts.

The fixed-query pattern in Q-Former and Perceiver Resampler designs cross-attends to a potentially large sequence of visual features. It converts variable-length visual input into a configured fixed-size token budget, whose retained evidence still needs task-specific evaluation.

Audio integration

So far we've focused on vision, but the same encoder-connector-LLM pattern applies to audio. Audio processing requires translating continuous sound waves into dense time-frequency features or learned embeddings. While the core architecture follows the same pattern, audio introduces unique challenges around time resolution and frequency representation.

Speech-heavy pipelines often start from log-Mel spectrograms, which is the path Whisper uses^{[10]Reference 10Whisper: Robust Speech Recognition via Large-Scale Weak Supervision.https://arxiv.org/abs/2212.04356}. Other audio encoders learn a waveform frontend directly instead of hand-specifying a spectrogram. For data-center alarms, fan noise, relay clicks, or other non-speech signals, contrastive models like CLAP (Contrastive Language-Audio Pretraining)^{[11]Reference 11CLAP: Learning Audio Concepts from Natural Language Supervision.https://arxiv.org/abs/2206.04769} are more appropriate.

Once audio is encoded into feature vectors, a connector can map those representations into prefix-token space or expose them as separate memory for cross-attention. Because audio can be lengthy, temporal compression (such as strided convolutions or pooling layers) can reduce sequence length before downstream fusion:

Practical considerations

Moving a multimodal architecture into production changes the workload shape. Vision and audio frontends add upstream work, fusion adds new memory paths, and variable evidence lengths complicate batching.

Latency & throughput

Visual encoders add work before the first output token can be generated, with cost depending on resolution, crop count, and batch policy. For some single-image routes that encoder pass dominates time to first token (TTFT); for others, visual prefill or decode dominates. Multiple high-resolution images or video frames increase the need to measure each stage.

Separate vision-encoding and text-generation pools can let each workload use different batching and autoscaling controls. That split also introduces RPC, cache-consistency, and scheduling costs, so it's a design to benchmark rather than a default latency win.

Memory bandwidth (KV cache)

In early-fusion systems, visual tokens consume KV cache (Key-Value cache) just like text tokens, so a 1,000-token visual prefix reduces the space available for text and output reservations. Cross-attention systems move more cost into separate visual memory and extra attention blocks instead of the main prompt prefix. Either way, memory pressure grows with batch size and admitted visual evidence.

To control this memory pressure, a serving system can use PagedAttention to allocate non-contiguous KV-cache blocks for long prompts, including prompts with large visual prefixes^{[18]Reference 18Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}. If a user asks multiple questions about the same image, cached encoder features or projected tokens avoid upstream repeated work; decoder prefill is skipped only when prefix/KV reuse is supported for the matching visual prefix.

Training instability

Balancing optimization across modalities during end-to-end or partially unfrozen training can be difficult. There often isn't a clean "vision loss" fighting a separate "text loss." Multimodal batches, connector updates, or modality-specific learning rates can drag the language model away from its pre-trained distribution, showing up as weaker pure-text behavior.

A possible contributor is mismatch between image features and decoder input states early in training, often called the modality gap. If the connector isn't aligned adequately, grounding quality can suffer. Track learning rates, gradients, pure-text regressions, and grounded task slices rather than assuming one symptom has one cause.

Safety & adversarial attacks

Visual inputs introduce attack vectors that text-only guardrails can't catch. A screenshot, PDF, or image can contain prompt injection text that the OCR or vision stack faithfully reads. The model isn't "cheating" here. It's doing exactly what it was trained to do: extract and follow visible instructions.

Adversarial or misleading visuals can also change model behavior even when raw pixels look benign to a human viewer. A production threat model should cover image moderation where needed, OCR-derived instruction isolation, and downstream policy checks on final actions or answers.

Reasoning after perception

Giving the decoder more reasoning budget can help on charts, diagrams, and long documents, but it's orthogonal to multimodal architecture. Extra compute only helps after perception succeeds. If the visual encoder missed a label or the projector discarded detail, more decoding steps won't bring that signal back.

Grounding evaluation should reflect that boundary: a fluent answer doesn't pass when the cited region or OCR evidence is absent.

Mastery check

What strong answers show

• Foundational: Map the flow of data through modality-specific encoders, connectors, and the LLM backbone.
• Intermediate: Choose early fusion, cross-attention, or late fusion for a concrete task and defend why.
• Advanced: Distinguish two-stage alignment pipelines from fully end-to-end multimodal training.
• Advanced: Apply token budget management through queried reduction, tiling, and temporal compression.
• Advanced: Analyze serving bottlenecks such as encoder latency, visual-prefix KV-cache growth in early-fusion designs, and where MoE helps or doesn't help.
• Advanced: Define a multimodal case API, size one evidence path, recover modalities independently, and roll feature generations back without mixing versions.

Skill levels to aim for

• Strong: You can trace encoder -> connector -> fusion choice -> serving budget for a concrete system and defend each tradeoff.
• Good: You can explain core components and choose a plausible fusion strategy, but may miss one training or serving consequence.
• Needs work: You still treat multimodal design as token concatenation, or you reach for post-training before fixing perception and token-budget problems.

Follow-up questions

Common pitfalls

"Multimodal design is mostly token concatenation"

• Symptom: The system can ingest image or audio tokens, but grounded answers stay shallow or brittle.
• Cause: Encoder, connector, and fusion choices were treated as wiring details instead of the real architecture.
• Fix: Design the full path explicitly: which encoder preserves evidence, how the connector compresses or exposes it, and how the LLM attends back to it.

"Hallucinated detail means the LLM needs more tuning"

• Symptom: The model invents labels, counts, or chart details even though the training data is clean.
• Cause: Evidence was lost earlier in the encoder or connector, so decoding is guessing from a weak representation.
• Fix: Debug in order: encoder resolution, connector bottleneck, alignment data, then post-training.

"Late fusion is fine for grounded generation"

• Symptom: The system scores images and text well but fails when it has to explain or point to supporting evidence.
• Cause: Late fusion keeps modalities too separate for token-level grounded generation.
• Fix: Use early fusion or cross-attention when the answer must stay tied to visual or audio evidence during decoding.

"Token budget only matters for text"

• Symptom: Prefill latency and KV-cache growth explode after adding high-resolution images or long audio clips.
• Cause: Visual and audio prefixes were treated as free context before the LLM.
• Fix: Budget multimodal tokens first. Use tiling, queried reduction, sparse sampling, or temporal compression before expanding the backbone.

"Preference tuning can recover missing perception"

• Symptom: Teams keep tuning policy behavior while OCR, chart reading, or grounding accuracy stays flat.
• Cause: Training is targeting the decoder after the evidence path has already failed.
• Fix: Fix encoder and connector quality first, then use post-training for abstention, policy alignment, or output style.

"MoE solves high-resolution multimodal cost by itself"

• Symptom: Backbone FLOPs drop, but multimodal serving is still slow and memory-heavy.
• Cause: MoE only sparsifies part of the backbone; it doesn't reduce encoder cost or long multimodal prefixes.
• Fix: Treat MoE as a backbone-capacity tool. Handle image, video, and audio token budgets separately.

Common design scenarios

Two system design scenarios show up frequently in AI engineering work:

Self-check

Use these checkpoints to verify that the architecture choices are now automatic.

Multimodal architecture checklist

1. The core decomposition is Modality Encoder -> Connector -> Language Backbone.
2. Connector choice determines how much visual detail survives and whether evidence becomes a projected prefix or a separate memory bank. Simple MLPs work well for direct projection, while Q-Former and Perceiver-style reducers keep downstream visual counts bounded.
3. Early fusion is common for generative VLMs. Cross-attention is useful when you want to keep visual features separate. Late fusion fits retrieval and classification more than grounded generation.
4. Two-stage alignment plus instruction tuning is a documented adapter-stack recipe. Published or disclosed native multimodal models train more of the multimodal path together. Compare task quality, compute, and regressions; preference tuning is optional, not universal.
5. The biggest serving bottlenecks are encoder latency, visual-feature prefill cost, and, in early-fusion designs, KV cache growth. Cache visual features and manage memory carefully.
6. MoE can reduce active FLOPs, but it doesn't solve visual token explosion by itself.

Treating multimodal input as just "text with extra tokens" is the recurring failure mode. Concatenation is easy compared with deciding what to encode, what to compress, and what the LLM should attend to.