Multimodal LLM Architecture

Vision-language models (VLMs) and CLIP (Contrastive Language-Image Pre-training)^[1] showed how image-text alignment and visual token budgets work. Multimodal large language model (LLM) architecture generalizes that into a product stack: modality encoders, projector design, fusion strategy, training recipe, serving cache, and safety boundary.

This design chapter explains how encoders, projection layers, shared representations, and product constraints align text, images, audio, and other signals into one reasoning system.

Imagine asking a support model to inspect a damaged-package photo, read the attached chat transcript, and use both to decide whether the customer needs a replacement or a return label. This is the fundamental challenge AI models face when trying to understand the world. Text, images, and audio are effectively different "languages": a pixel has no obvious relationship to a word.

For a model to answer from those inputs, it needs a trained interface between modality features and its language path. This is the core of multimodal large language model (LLM) design: deciding what visual or audio evidence is encoded, what is compressed, and how the decoder can use it.

The concept: one shared case file

Think of a multimodal model like an operations desk that merges signals from several systems into one case file.

• The Text Path turns chat messages, policy snippets, and order notes into token embeddings (using a tokenizer) that the language model can reason over.
• The Vision Encoder turns package photos, product images, or warehouse snapshots into dense visual features.
• The Connector maps or exposes those visual features to the language path, through projected prefix tokens, cross-attention memory, or another published interface.

Another way to picture this is evidence routing. The vision encoder turns image evidence into feature records. The language model already works with text token states. A connector makes image features consumable by the language path; it does not automatically prove that "damaged box" pixels and words are aligned or that the answer is grounded. Alignment and grounding must be trained and evaluated.

Architecture components

Adapter-based vs jointly trained stacks

Published multimodal systems include two useful patterns. Adapter-based models keep a strong modality encoder and text LLM, then learn a connector between them. BLIP-2 (Bootstrapping Language-Image Pre-training, version 2)^[2] and LLaVA (Large Language-and-Vision Assistant)^[3] are examples. Jointly trained multimodal stacks, such as PaLI (Pathways Language and Image)^[4] and Gemini^[5], optimize more of the multimodal stack together on multimodal mixtures.

For concrete published or disclosed examples: Llama 4 describes early fusion over interleaved text, image, and video data^[6]; OpenAI describes GPT-4o as trained end-to-end across text, vision, and audio^[7]; Qwen2.5-VL describes a dynamic-resolution vision encoder trained with its language stack.^[8] These reports establish architecture families, not a universal ranking over adapter-based and joint approaches.

Adapter-based designs let a team reuse pretrained components and train fewer parameters. Joint training exposes more parameters and modalities to optimization, raising data, compute, and stability requirements. Choose from measured target-task quality, budget, and controllability rather than assuming one family wins.

Modality-specific encoders

Before the language model can process non-text inputs, each modality requires its own dedicated encoder to extract meaningful, high-level features. These encoders compress raw signals (such as noisy RGB pixels or high-frequency audio waveforms) into dense mathematical representations that capture semantic information.

Here's how different modality inputs flow through their respective encoders:

Key observations

• Pre-trained vs. Scratch: Pre-trained encoders (e.g., CLIP^[1], SigLIP^[9]) are practical starting points when their visual domain overlaps your task.
• Frozen vs. Fine-tuned: Freezing the encoder initially is one stable alignment recipe; later unfreezing is an experiment with quality and forgetting risks.
• Backbone Selection: CLIP ViT and SigLIP are standard starting points for image encoders. For audio, Whisper^[10] is common for speech-heavy pipelines, while CLAP (Contrastive Language-Audio Pretraining)^[11] is better when the goal is general audio-text alignment rather than transcription.

Projection layers

Non-text modalities produce feature tensors that do not directly match the decoder interface. In a projected-prefix design, a projector maps features into token-shaped states that enter the decoder context. In a cross-attention design, a connector can instead expose a separate visual memory bank.

The connector determines what evidence the decoder receives. If this step throws away spatial detail or its alignment training fails, the decoder may sound fluent while grounding poorly in the input.

A projector can be as simple as a linear layer or as structured as a learned bottleneck with cross-attention. The runnable example below models the systems consequence: some projectors keep every visual token, while Q-Former and Perceiver-style reducers bound the number sent to the LLM.

Projection methods compared

Q-Former and Perceiver-style resamplers address the same systems problem: bound visual features before the language path, with potential information loss to evaluate. BLIP-2^[2] uses a Q-Former. Flamingo^[12] uses a Perceiver Resampler before gated cross-attention layers.

The token count problem

A 224×224 image processed by a ViT-L/14 (Vision Transformer Large with a patch size of 14) encoder produces a 16×16 patch grid, or 256 patch tokens. Some implementations also keep a CLS token, yielding 257 total encoder tokens. Raise the resolution and the count climbs fast: LLaVA-1.5 swapped in a 336px CLIP encoder, which yields a 24×24 grid and 576 visual tokens per image.^[13] In an early-fusion path, admitting thousands of visual tokens increases shared self-attention work, prefix length, and KV-cache demand.

Worked example: how many tokens does one image cost?

Let's walk through the arithmetic so you can do this in an interview or on a whiteboard.

• Input: a single 224×224 image.
• Encoder: ViT-L/14, which means the patch size is 14 pixels.
• Step 1: Divide the image width by the patch size: $224 \div 14 = 16$ patches across.
• Step 2: The height gives the same count, so the total patches are $16 \times 16 = 256$.
• Result: The image becomes 256 visual tokens. If your LLM context window is 4096 tokens, one image alone eats roughly 6% of your total budget. Add a second image and you've spent 12%. Add a one-minute video at two frames per second and you're looking at 30,720 tokens, far beyond most context windows.

This is why visual token budgeting is a first-class design decision in architectures that admit visual tokens to the decoder or repeatedly attend over visual memory.

Solutions

1. Spatial compression: Pooling or resampling visual tokens (e.g., 256 → 64).
2. Temporal compression: Sampling keyframes or using temporal attention layers (e.g., TimeSformer (Time-Space Transformer)^[14], VideoMAE (Video Masked Autoencoders)^[15]).
3. Dynamic resolution: Tiling or cropping high-resolution inputs so detailed regions get more compute than the rest of the image.

Fusion strategies

Compare where text and visual features meet. Early fusion mixes tokens into one sequence before the transformer. Cross-attention lets configured language layers read visual memory. Late fusion pools each side independently and only merges at the end. None of these placements guarantees grounding; each defines what evidence and serving cost the model can incur.

Early fusion (visual prefix or interleaving)

In early fusion, projected modality tokens are inserted directly into the model input sequence, usually as a contiguous visual prefix or at special placeholder positions inside the prompt. After the projector creates decoder-compatible token states, the model treats image tokens and text tokens as one long sequence and runs standard self-attention over the whole thing. This is used in LLaVA-style architectures and published native multimodal transformers such as Llama 4:

Pros

Provides token-level image-text interaction through shared self-attention from the first decoder layer that receives the prefix.

Cons

Computationally expensive for long visual sequences, as image tokens consume a significant portion of the shared context window capacity.

Late fusion

Late fusion is common in dual-encoder retrieval or classification systems. Each modality is encoded mostly independently, pooled into a compact embedding, and merged by a lightweight head. This design supports indexed scoring, but cannot by itself produce answers grounded in detailed visual tokens because a generative decoder is absent from that path:

Pros

Highly modular for scoring. Encoders can be swapped independently, and a retrieval path avoids a long generative multimodal prefix.

Cons

Insufficient by itself for grounded generation. Because the scoring path never presents token-level image evidence to a decoder, optical character recognition (OCR)-heavy answering or region-cited reasoning needs an additional model path.

Cross-attention fusion (Flamingo-style)

Cross-attention layers allow text states to compute context-dependent weights over visual features at configured model depths, as seen in Flamingo.^[12] The code below demonstrates a gated cross-attention layer that takes the LLM's hidden text state and frozen visual features as inputs, returning a fused representation. Flamingo initializes the gate at zero so training begins from the language-only path and can learn to admit visual information:

$$\text{Attention}(Q_{\text{text}}, K_{\text{vis}}, V_{\text{vis}}) = \text{softmax}\left(\frac{Q_{\text{text}} K_{\text{vis}}^T}{\sqrt{d_k}}\right) V_{\text{vis}}$$

Architecture

Gated cross-attention is injected between standard LLM layers to incorporate frozen visual features:

At initialization, a zero gate leaves the language path unchanged by the visual residual. Training can then learn non-zero visual contributions.

Compared with early fusion, this keeps dense visual tokens out of the main decoder prefix. It does not make vision free: every configured cross-attention block still performs extra projections and attention against visual memory.

Training strategies

Two-stage training (LLaVA protocol)

The standard protocol for visual instruction tuning, popularized by LLaVA (Large Language-and-Vision Assistant)^[3] and refined in LLaVA-1.5^[13]:

Stage 1: Feature Alignment

• Freeze: Vision encoder + LLM
• Train: Projection layer only
• Data: ~595K image-text pairs (e.g., CC3M (Conceptual Captions 3M)^[16], COCO (Common Objects in Context)^[17] filtered)
• Objective: Align visual features with LLM word embeddings.

Stage 2: Visual Instruction Tuning

• Freeze: Vision encoder (in the reported LLaVA recipe)
• Train: Projection layer + LLM (full fine-tune or LoRA)
• Data: ~158K multimodal instruction data (complex reasoning, generated via GPT-4)^[3].
• Objective: Learn to follow instructions and reason about visual content.

Those counts describe original LLaVA recipe: a filtered CC3M pretraining set plus 158K GPT-4-generated multimodal instructions^[3]. LLaVA-1.5 kept same two-stage shape, but swapped in a 336px CLIP encoder, an MLP connector, and a larger public task mixture. Its paper reports roughly 1.2M total training examples and full training in about one day on a single 8-A100 node^[13].

For custom domains like damaged-package inspection, product catalog QA, or complex charts, compare a connector-alignment stage against direct instruction tuning on held-out grounding slices. Stage ordering is an experimental choice outside the reported LLaVA recipe, not a guaranteed improvement.

Here's how the two stages connect:

End-to-end training

While two-stage training keeps the pre-trained components largely frozen, end-to-end training updates the modality encoder, projector, and core transformer together. Jointly trained families such as PaLI^[4], Gemini^[5], and Llama 4^[6] represent this direction and aim for a more native multimodal representation. Llama 4 warm-starts from a MetaCLIP-based vision encoder that was first aligned against a frozen Llama, then trains the whole stack on interleaved data; in this published recipe, joint training does not mean starting every component from random weights^[6].

This strategy allows gradients to update cross-modal representations jointly. Instead of holding the vision encoder fixed, training can adapt its features to downstream objectives. Whether this improves chart reading, OCR, or grounding depends on data, optimization, and evaluation.

However, joint training exposes more parameters and objectives to optimization. It requires substantial compute and balanced datasets containing text and multimodal examples. Multimodal updates can conflict with language-modeling behavior, so text-only regression measurement remains part of the training contract.

Trade-offs

• Compute: Higher training cost than a projector-only stage because gradients update the encoder and language model as well.
• Stability: Harder to stabilize; carefully tuned learning rates and loss weighting are required to prevent one modality from dominating.
• Quality: Can adapt the encoder to downstream objectives; measure gains against adapter baselines on target slices.

Post-training for grounded behavior

After instruction tuning, many teams add a post-training stage focused on response quality, abstention, and safety. Direct Preference Optimization (DPO)^[18] is one option, but it's not a universal third stage for multimodal systems. Depending on the product, teams may use supervised preference data, rejection sampling, RLHF, or task-specific evaluation loops instead.

The important systems point is that post-training can improve how the model explains uncertainty, refuses unsupported claims, and follows product policy. It can't recover information that was already lost upstream. If the vision encoder missed small text or the projector over-compressed the image, no preference method can reconstruct that detail later.

If a multimodal model hallucinates image details, inspect encoder resolution, connector bottleneck, alignment data, and post-training separately. If required evidence never reaches the decoder, preference tuning cannot reconstruct it.

Efficient inference: token budget before parameter budget

Multimodal latency includes modality encoding and processing visual features inside the decoder. In early-fusion systems, that second cost can show up as a large prefill over visual tokens. In cross-attention systems, the prefix stays smaller, but each fusion block still attends over visual memory. Measure these components by route before choosing whether to compress inputs or optimize decode. PagedAttention-style serving makes large, irregular KV caches easier to manage under batching pressure^[19].

If users ask multiple questions about the same image, caching encoder features avoids rerunning the vision frontend. Caching projected tokens avoids repeating projection. Only a compatible prefix/KV-cache reuse mechanism avoids repeating decoder prefill over an identical visual prefix.

Where MoE helps, and where it doesn't

Mixture-of-Experts (MoE) layers activate only a subset of experts per token, which lowers active FLOPs at a given parameter count^[20]. That can improve throughput for very large multimodal backbones, but it doesn't remove the main multimodal bottlenecks:

• Vision or audio encoders still have to run.
• In early-fusion systems, long visual prefixes still bloat the KV cache.
• High-resolution inputs still create expensive prefills.

Treat MoE as a model-capacity tool, not as the first fix for high-resolution image serving.

Handling variable-length visual tokens

A major bottleneck in multimodal architectures is managing the explosion of tokens when processing high-resolution images or long videos. Keeping every patch token from every crop or frame is computationally prohibitive because self-attention scales quadratically with sequence length. Two prominent strategies exist to control this budget while preserving important details.

Tiling and multi-crop encoding

One common way to preserve high-resolution detail is to encode a global thumbnail plus a small set of crops chosen by a layout, OCR, or task policy. The thumbnail preserves overall layout, while selected crops can preserve fine text, tables, or localized objects. Crop selection is itself an evaluated component: a skipped region cannot be recovered downstream.

The trade-off is straightforward. More crops improve OCR and small-object reasoning, but every crop adds another encoder pass and another batch of visual tokens. That makes this strategy best for documents, charts, maps, and other inputs where fine detail matters.

Newer native multimodal models push this further. Qwen2.5-VL processes images at their native resolution with a dynamic-resolution ViT, then merges each 2×2 patch block into one token so the visual token count tracks actual image size instead of a fixed grid^[8]. The systems goal is the same as tiling: spend visual tokens where detail lives, and keep the budget bounded everywhere else.

Because the number of crops depends on aspect ratio and resolution, the resulting token sequence length is variable. A wide panorama might need a horizontal strip of crops, while a long document might need a vertical stack:

Queried reduction (Perceiver / Q-Former^[2])

This approach (popularized by models like Flamingo^[12] and BLIP-2^[2]) uses a fixed set of learned query vectors to scan the visual features and compress them into a predefined token count. By decoupling the visual input resolution from the sequence length passed to the language model, this architecture helps prevent out-of-memory failures and keeps the effective visual token budget bounded even for high-framerate video or dense spatial layouts.

The fixed-query pattern in Q-Former and Perceiver Resampler designs cross-attends to a potentially large sequence of visual features. It converts variable-length visual input into a configured fixed-size token budget, whose retained evidence still needs task-specific evaluation.

Audio integration

So far we've focused on vision, but the same encoder-projector-LLM pattern applies to audio. Audio processing requires translating continuous sound waves into dense time-frequency features or learned embeddings that can be projected into the same token space as text. While the core architecture follows the same encoder-projector-LLM pattern, audio introduces unique challenges regarding time resolution and frequency representation.

Speech-heavy pipelines often start from log-Mel spectrograms, which is the path Whisper uses^[10]. Other audio encoders learn a waveform frontend directly instead of hand-specifying a spectrogram. For warehouse alarms, conveyor sounds, delivery-door audio, or other non-speech signals, contrastive models like CLAP (Contrastive Language-Audio Pretraining)^[11] are more appropriate.

Once the audio is encoded into feature vectors, an audio projection layer maps these continuous representations into the LLM's token embedding space. Because audio can be lengthy, temporal compression (such as strided convolutions or pooling layers) is frequently applied within the projector to reduce the sequence length before the tokens are interleaved with text:

Practical considerations

When moving from a theoretical multimodal architecture to a production system, engineering teams face significant infrastructure and alignment challenges. Simply adding vision or audio encoders to an LLM changes its performance characteristics entirely.

Latency & throughput

Visual encoders add work before the first output token can be generated, with cost depending on resolution, crop count, and batch policy. For some single-image routes that encoder pass dominates time-to-first-token; for others, visual prefill or decode dominates. Multiple high-resolution images or video frames increase the need to measure each stage.

Separate vision-encoding and text-generation pools can let each workload use different batching and autoscaling controls. That split also introduces RPC, cache-consistency, and scheduling costs, so it is a design to benchmark rather than a default latency win.

Memory bandwidth (KV cache)

In early-fusion systems, visual tokens consume KV cache (Key-Value cache) just like text tokens, so a 1,000-token visual prefix reduces the space available for text and output reservations. Cross-attention systems move more cost into separate visual memory and extra attention blocks instead of the main prompt prefix. Either way, memory pressure grows with batch size and admitted visual evidence.

To mitigate this, a serving system can use memory optimization techniques. PagedAttention helps manage fragmented KV-cache memory by allocating non-contiguous blocks for long prompts, including prompts with large visual prefixes^[19]. Additionally, if a user asks multiple questions about the same image, cached encoder features or projected tokens avoid upstream repeated work; decoder prefill is skipped only when prefix/KV reuse is supported for the matching visual prefix.

Training instability

Balancing optimization across modalities during end-to-end or partially unfrozen training can be difficult. There often is not a clean "vision loss" fighting a separate "text loss." Multimodal batches, connector updates, or modality-specific learning rates can drag the language model away from its pre-trained distribution, showing up as weaker pure-text behavior.

A possible contributor is mismatch between image features and decoder input states early in training, often called the modality gap. If the connector is not aligned adequately, grounding quality can suffer. Track learning rates, gradients, pure-text regressions, and grounded task slices rather than assuming one symptom has one cause.

Safety & adversarial attacks

Visual inputs introduce attack vectors that text-only guardrails can't catch. A screenshot, PDF, or image can contain prompt injection text that the OCR or vision stack faithfully reads. The model isn't "cheating" here. It's doing exactly what it was trained to do: extract and follow visible instructions.

Adversarial or misleading visuals can also change model behavior even when raw pixels look benign to a human viewer. A production threat model should cover image moderation where needed, OCR-derived instruction isolation, and downstream policy checks on final actions or answers.

Reasoning after perception

Giving the decoder more reasoning budget can help on charts, diagrams, and long documents, but it's orthogonal to multimodal architecture. Extra compute only helps after perception succeeds. If the visual encoder missed a label or the projector discarded detail, more decoding steps won't bring that signal back.

Grounding evaluation should reflect that boundary: a fluent answer does not pass when the cited region or OCR evidence is absent.

Mastery check

By the end of this lesson, you should be able to:

• Foundational: Map the flow of data through modality-specific encoders, projector layers, and the LLM backbone.
• Intermediate: Choose early fusion, cross-attention, or late fusion for a concrete task and defend why.
• Advanced: Distinguish two-stage alignment pipelines from fully end-to-end multimodal training.
• Advanced: Apply token budget management through queried reduction, tiling, and temporal compression.
• Advanced: Analyze serving bottlenecks such as encoder latency, visual-prefix KV-cache growth in early-fusion designs, and where MoE helps or does not help.

Evaluation rubric

• Strong: You can trace encoder -> projector -> fusion choice -> serving budget for a concrete product and defend each tradeoff.
• Good: You can explain core components and choose a plausible fusion strategy, but may miss one training or serving consequence.
• Needs work: You still treat multimodal design as token concatenation, or you reach for post-training before fixing perception and token-budget problems.

Follow-up questions

Common pitfalls

"Multimodal design is mostly token concatenation"

Symptom: The system can ingest image or audio tokens, but grounded answers stay shallow or brittle. Cause: Encoder, projector, and fusion choices were treated as wiring details instead of the real architecture. Fix: Design the full path explicitly: which encoder preserves evidence, how the projector compresses it, and how the LLM attends back to it.

"Hallucinated detail means the LLM needs more tuning"

Symptom: The model invents labels, counts, or chart details even though the training data is clean. Cause: Evidence was lost earlier in the encoder or projector, so decoding is guessing from a weak representation. Fix: Debug in order: encoder resolution, projector bottleneck, alignment data, then post-training.

"Late fusion is fine for grounded generation"

Symptom: The system scores images and text well but fails when it has to explain or point to supporting evidence. Cause: Late fusion keeps modalities too separate for token-level grounded generation. Fix: Use early fusion or cross-attention when the answer must stay tied to visual or audio evidence during decoding.

"Token budget only matters for text"

Symptom: Prefill latency and KV-cache growth explode after adding high-resolution images or long audio clips. Cause: Visual and audio prefixes were treated as free context before the LLM. Fix: Budget multimodal tokens first. Use tiling, queried reduction, sparse sampling, or temporal compression before expanding the backbone.

"Preference tuning can recover missing perception"

Symptom: Teams keep tuning policy behavior while OCR, chart reading, or grounding accuracy stays flat. Cause: Training is targeting the decoder after the evidence path has already failed. Fix: Fix encoder and projector quality first, then use post-training for abstention, policy alignment, or output style.

"MoE solves high-resolution multimodal cost by itself"

Symptom: Backbone FLOPs drop, but multimodal serving is still slow and memory-heavy. Cause: MoE only sparsifies part of the backbone; it does not reduce encoder cost or long multimodal prefixes. Fix: Treat MoE as a backbone-capacity tool. Handle image, video, and audio token budgets separately.

Common design scenarios

Two critical system design scenarios show up frequently in AI engineering work:

Self-check

Use these checkpoints to verify that the architecture choices are now automatic.

Key takeaways

1. Architecture: The standard recipe is Modality Encoder → Projection Layer → LLM.
2. Projection: Projector choice determines how much visual detail survives. Simple MLPs work well, while Q-Former and Perceiver-style reducers keep token counts bounded.
3. Fusion: Early fusion is common for generative VLMs. Cross-attention is useful when you want to keep visual features separate. Late fusion fits retrieval and classification more than grounded generation.
4. Training: Two-stage alignment plus instruction tuning is a documented adapter-stack recipe. Published or disclosed native multimodal models train more of the multimodal path together. Compare task quality, compute, and regressions; preference tuning is optional, not universal.
5. Serving: Biggest bottlenecks are encoder latency, visual-feature prefill cost, and, in early-fusion designs, KV cache growth. Cache visual features and manage memory carefully.
6. Scaling: MoE can reduce active FLOPs, but it doesn't solve visual token explosion by itself.

Treating multimodal input as just "text with extra tokens" is the recurring failure mode. The hard part is not concatenation. It is deciding what to encode, what to compress, and what the LLM should actually attend to.