Multimodal LLM Architecture

The copilot gets asked, “Did p95 latency rise after the 14:02 deploy?” It sees a Grafana screenshot and an incident-bridge transcript, then cites a chart spike from 14:02 alongside a sentence spoken at 14:22. Both details are real. The answer is still wrong because the path lost modality alignment: evidence survived individually, but not its relationship.

CLIP (Contrastive Language-Image Pre-training)^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} showed how to align a rack photo with a text query, and why visual token budgets get expensive once you stop pooling the image into one vector. That still doesn't give you a system that can look at a Grafana screenshot, listen to the incident bridge, and cite both before it names a deploy, a quota, or a dependency failure.

Pixels, waveforms, and words don't share a native interface. A large language model (LLM) can reason over dashboard evidence only after a trained path encodes it, compresses it, and exposes it in a form the decoder was trained to read. If that path drops an axis label, later decoding can't invent the missing ticks honestly.

Before choosing a model family, write down what must survive: chart coordinates, transcript timestamps, and the boundary between evidence and instructions. Each later component gets judged against that contract.

The incident-evidence copilot is that path: encoders, connectors, fusion, training, serving cache, and the safety boundary around image-derived text.

Product brief: an incident-evidence copilot

Start with the answer contract. Every claim needs a source span, region, or tool result; when required evidence is missing, the system must choose partial, abstained, or review instead of guessing. That choice drives the API and the rest of the design.

Design a copilot that accepts a dashboard screenshot, a call transcript or audio clip, and an operator question. It returns a diagnosis only when it can cite the text span, chart region, or tool result that supports the answer.

Requirements and API

• POST /v1/cases accepts immutable image, audio, and transcript references plus tenant, incident, and idempotency keys. It returns case_id and per-modality processing state.
• POST /v1/cases/{case_id}/questions accepts question and a latency class. It returns answer, status (grounded, partial, abstained, or review), evidence IDs with text spans or image regions, and encoder, connector, model, and policy versions.
• Interactive questions target p95 below 3 seconds for one image and an existing transcript. Audio transcription and long-video processing may finish asynchronously.
• The system must isolate tenants, treat image-derived instructions as untrusted evidence, and refuse claims whose cited region or transcript span is missing.

Data flow and sizing

Follow one case through the path before sizing models. Admission owns the original bytes, frontends own modality-specific representations, and the connector owns the boundary where those representations become readable to the language path.

The path is artifact admission -> format and policy checks -> vision/audio/text frontends -> versioned feature store -> connector or reducer -> fusion and language model -> evidence validator -> policy gate -> answer trace. Raw artifacts stay immutable. Each derived feature records its source hash and model version.

Admission decides which bytes are allowed in. Frontends turn those bytes into features. The connector is the first place the language path can actually read them.

Now ask which capacity limit fails first: visual tokens, transcript tokens, or output reserve. The arithmetic below turns that question into an admission decision.

Suppose one dashboard uses a global thumbnail plus four crops. A direct Vision Transformer (ViT) path at 576 patch tokens per view would create 5 x 576 = 2,880 visual tokens. A tested reducer that emits 32 tokens per view lowers that to 160 before adding a 2,000-token transcript and a 300-token output reservation. On a 4,096-token early-fusion route that reserve leaves 4,096 - 300 = 3,796 tokens for prefix. The uncompressed mix (2,880 + 2,000 = 4,880) doesn't fit; the reduced mix (160 + 2,000 = 2,160) does. At 20 queries per second (QPS), use separate encoder and decoder pools so bursty image work, cached-case follow-ups, and audio backfills can scale independently.

Recovery, rollout, and evaluation

A route that fits one request can still fail when a modality retries or degrades. Decide in advance which missing states allow a labeled partial answer and which force abstention, then make those states observable.

Each modality stage is idempotent and records pending, ready, failed, or quarantined. A failed image encoder can retry without rerunning completed transcription. If one modality is unavailable, the API may return a clearly labeled partial answer only when product policy permits it; otherwise it abstains. Never silently substitute language priors for missing visual evidence.

Roll out one modality and connector version at a time: frozen offline cases, shadow comparison, a tenant-limited canary, then staged expansion with the old feature generation available for rollback. Gate on answer faithfulness, optical character recognition (OCR) exact match, region-citation IoU (intersection over union), transcript word error rate, abstention quality, prompt-injection tests, p95 latency, GPU memory, and cost per grounded answer. Slice results by modality combination, evidence length, image resolution, audio quality, tenant, and failure path.

A fluent draft isn't a release. The evidence gate can still send that draft to abstained or review.

The concept: one shared evidence path

Before naming components, predict what the decoder must receive from each source. It needs text states for the question and transcript, visual states that retain regions and labels, and audio states that retain the time or event needed for attribution.

A multimodal system must turn dashboard pixels, transcript spans, and audio features into representations that one large language model (LLM) was trained to consume, while preserving where each piece of evidence came from.

• The Text Path turns incident messages, runbook snippets, and trace summaries into token embeddings (using a tokenizer) that the language model can reason over.
• The Vision Encoder turns dashboard screenshots, chart crops, and warning badges into dense visual features.
• The Connector maps or exposes those visual features to the language path, through projected prefix tokens, cross-attention memory, or another published interface.

The vision encoder turns image evidence into feature records. The language model already works with text token states. A connector makes image features consumable by the language path; it doesn't automatically prove that "axis label" pixels and words are aligned or that the answer is grounded. Alignment and grounding must be trained and evaluated.

Why can't a text LLM reason over pixels directly?

Answer

Pixels are raw grid values rather than decoder-ready features. A modality encoder produces visual representations, and a connector exposes them through projected prefix tokens or cross-attention memory in a form the language path was trained to consume.

That decomposition leaves one interface decision: should visual evidence join the text prefix, or remain a side memory that selected layers read? The next sections make that choice measurable.

Encoders, connector, decoder

The encoder isn't the answer generator. It decides which evidence is available to the bridge, so a lost axis label or timestamp can't be recovered by a stronger decoder later.

Adapter-based vs jointly trained stacks

Once the evidence path is clear, choose where cross-modal alignment is learned. An adapter stack concentrates learning in the connector; a joint stack lets the modality encoder and language backbone move together.

Published multimodal systems include two useful patterns. Adapter-based vision-language models (VLMs) keep a strong modality encoder and a text LLM, then learn a connector between them. BLIP-2 (Bootstrapping Language-Image Pre-training, version 2)^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} and LLaVA (Large Language-and-Vision Assistant)^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485} are examples. Jointly trained multimodal stacks, such as PaLI (Pathways Language and Image)^{[4]Reference 4PaLI: A Jointly-Scaled Multilingual Language-Image Model.https://arxiv.org/abs/2209.06794} and Gemini^{[5]Reference 5Gemini: A Family of Highly Capable Multimodal Models.https://arxiv.org/abs/2312.11805}, optimize more of the multimodal stack together on multimodal mixtures.

For concrete published or disclosed examples: Llama 4 describes early fusion over interleaved text, image, and video data, with a MetaCLIP-based encoder first aligned against a frozen Llama, then joint training of the stack^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/}; OpenAI describes GPT-4o as one network trained end-to-end across text, vision, and audio, including generated speech and image outputs^{[7]Reference 7GPT-4o System Card.https://arxiv.org/abs/2410.21276}; Qwen2.5-VL describes a dynamic-resolution vision encoder trained with its language stack and a 2×2 patch merger that bounds visual tokens.^{[8]Reference 8Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923} These reports establish architecture families, not a ranking you can copy into the copilot without measuring grounding on your dashboards.

That last GPT-4o detail matters for product scope. The copilot consumes screenshots and audio and returns cited text. Generating pixels is a different generative path, covered next. Don't assume a model that reads a Grafana panel can emit a trustworthy replacement for it.

If reuse and narrow training scope dominate, adapters give you a smaller first experiment. If a new modality or domain needs the whole representation to move, joint training may be worth its cost. Either choice still needs the same grounding slices.

Adapter-based designs let a team reuse pretrained components and train fewer parameters. Joint training exposes more parameters and modalities to optimization, raising data, compute, and stability requirements. Choose from measured target-task quality, budget, and controllability rather than assuming one family wins.

When would you choose an adapter-based (bolt-on) stack over native joint training?

Answer

Choose adapters when reuse of frozen components and narrow training scope fit your budget, as in LLaVA. Evaluate a jointly trained system when you have suitable data and compute or consume one through an API; don't assume either architecture wins grounding without task-specific measurements.

Modality-specific encoders

A screenshot, a bridge recording, and a transcript carry different evidence. Before selecting backbones, ask which detail answers the operator's question: a chart coordinate, a spoken timestamp, or the words already in text form.

A practical design starts by asking how each non-text modality becomes task-useful features. In encoder-connector stacks, modality frontends compress raw signals (noisy RGB pixels, a 40-second bridge recording) into dense representations.

Those inputs don't share a feature rate. For the copilot's running numbers, five 336px dashboard views are 2,880 patch tokens, 40 seconds of audio at 100 Hz is 4,000 steps before stride, and the transcript is 2,000 tokens. The connector's job is to publish one decoder interface from that mess:

Encoder choices

Now choose each frontend by evidence type and domain overlap, then measure whether its representation keeps the detail the connector needs.

• Pretrained vs from scratch. CLIP^{[1]Reference 1Learning Transferable Visual Models From Natural Language Supervision.https://arxiv.org/abs/2103.00020} and SigLIP^{[9]Reference 9Sigmoid Loss for Language Image Pre-training.https://arxiv.org/abs/2303.15343} are practical image starting points when their visual domain overlaps your dashboards. Train an encoder from scratch only if that overlap is poor and you can afford the data.
• Frozen vs fine-tuned. Freeze the encoder while the connector learns the bridge. Unfreezing later is an experiment with quality gains and forgetting risk, not a default.
• Backbone by modality. CLIP ViT and SigLIP are standard image starts. For speech-heavy incident audio, Whisper^{[10]Reference 10Whisper: Robust Speech Recognition via Large-Scale Weak Supervision.https://arxiv.org/abs/2212.04356} is the usual transcription frontend. CLAP (Contrastive Language-Audio Pretraining)^{[11]Reference 11CLAP: Learning Audio Concepts from Natural Language Supervision.https://arxiv.org/abs/2206.04769} is the better fit when you need alarm or fan-noise alignment rather than a transcript.

Why start from pre-trained modality encoders?

Answer

They can provide useful existing visual or audio features when the domain overlaps pretraining. Freezing them early reduces the number of moving parts while a connector is aligned; domain shift may still require adaptation.

Connectors and projection layers

After frontends produce features, predict the failure at the boundary: a projector may preserve every patch but spend the whole context window, while a reducer may fit the budget and erase the tiny badge that proves the diagnosis.

Non-text modalities produce feature tensors that don't directly match the decoder interface. In a projected-prefix design, a projector maps features into token-shaped states that enter the decoder context. In a cross-attention design, a connector can instead expose a separate visual memory bank.

The connector determines what evidence the decoder receives. If this step throws away spatial detail or its alignment training fails, the decoder may sound fluent while grounding poorly in the input.

What failure mode points to the projector rather than the LLM decoder?

Answer

If answers are fluent but visually ungrounded, inspect encoder evidence and connector compression before only retuning the decoder. The decoder can reason only over detail exposed through its visual interface.

A connector can be as simple as a linear projector or as structured as a learned bottleneck with cross-attention. The runnable example below models the systems consequence: some connectors keep every visual feature, while Q-Former and Perceiver-style reducers bound the downstream visual stream before a prefix or cross-attention path exposes it.

For 576 visual features, a linear or MLP bridge leaves 576 model-visible states, while a 32-query reducer publishes 32. That is the capacity win; the question to test is whether the omitted 544 states contained evidence the copilot needed.

Connector methods compared

Q-Former and Perceiver-style resamplers address the same systems problem: bound visual features before the language path, with potential information loss to evaluate. BLIP-2^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597} uses a Q-Former. Flamingo^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} uses a Perceiver Resampler to produce a fixed visual bank before gated cross-attention layers.

Why do Q-Former and Perceiver-style reducers matter for serving?

Answer

They bound the number of visual tokens sent downstream. This can reduce prefill, context-window pressure, or visual-memory cost; evaluation must confirm that required image evidence survived compression.

Connector depth is another design choice

Once the token count is bounded, placement becomes another interface decision. A connector can join at the input, or expose modality state to selected decoder blocks later in the network.

An input-layer projector isn't the only place where a modality can join a decoder. A connector can expose image features to selected intermediate or late decoder blocks instead. This changes both the representation interface and the amount of model computation involved in adaptation.

MPnP is one concrete research design for runtime modality adaptation on resource-constrained devices.^{[13]Reference 13Modality Plug-and-Play: Runtime Modality Adaptation in LLM-Driven Autonomous Mobile Systemshttps://doi.org/10.1145/3680207.3723491} It makes two separate depth choices:

1. It extracts multi-level features from several intermediate vision-encoder blocks instead of relying on one final feature alone.
2. It maps those features into key-value pairs and injects them, with trainable per-block weights, into a selected set of the decoder's last blocks.

Moving an insertion point closer to the output shortens the backpropagation path during runtime adaptation. It also gives fewer language blocks an opportunity to combine modality evidence. MPnP therefore tunes the number of connected blocks as an accuracy-versus-training-cost choice. Its reported savings come from scoped runtime-training experiments with relatively small OPT and BLOOMZ backbones on mobile-system visual question-answering tasks. They don't establish that late-block injection is universally better, or that it reduces every production inference path.

What is the difference between tapping deeper vision features and injecting them into deeper language blocks?

Answer

The first choice decides which levels of visual representation the connector reads. The second decides where those representations become available inside the language model. They are independent choices and should be ablated separately.

The token count problem

Before comparing model sizes, predict the sequence the decoder will actually see. Resolution, view count, frame rate, patch size, and connector reduction can dominate the context and memory budget.

A 224×224 image processed by a ViT-L/14 (Vision Transformer Large with a patch size of 14) encoder produces a 16×16 patch grid, or 256 patch tokens. Some implementations also keep a CLS token, yielding 257 total encoder tokens. Raise the resolution and the count climbs fast: LLaVA-1.5 swapped in a 336px CLIP encoder, which yields a 24×24 grid and 576 visual tokens per image.^{[14]Reference 14Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744} In a direct projected-prefix path, admitting thousands of visual tokens increases shared self-attention work, prefill length, and KV-cache demand. A separate cross-attention bank has a different cost path.

Worked example: how many tokens does one image cost?

Make this estimate before you pick a fusion route. The arithmetic is short enough to do in an interview or on a whiteboard.

• Input: a single 224×224 image.
• Encoder: ViT-L/14, which means the patch size is 14 pixels.
• Step 1: Divide the image width by the patch size: $224 \div 14 = 16$ patches across.
• Step 2: The height gives the same count, so the total patches are $16 \times 16 = 256$.
• Result: The image becomes 256 visual tokens before compression. In a direct projected-prefix route with a 4096-token context window, one image alone eats roughly 6% of your total budget. Add a second image and you've spent 12%. Add a one-minute video at two frames per second and you're looking at 30,720 raw patch tokens, or 7.5 times that route's full allowance, before temporal compression.

Visual token budgeting is a first-class design decision in architectures that admit visual tokens to the decoder or repeatedly attend over visual memory.

A 224x224 image through ViT-L/14 produces how many patch tokens, and why does it matter?

Answer

$224 / 14 = 16$ patches per side, so $16 x 16 = 256$ patch tokens before any CLS token. If a direct projected-prefix connector preserves them all, those tokens consume context, prefill compute, and KV cache just like text tokens. A reducer or separate cross-attention bank changes the downstream cost path.

File size isn't a visual-token budget

Separate transport capacity from model capacity before setting limits. A smaller upload can still decode into a larger model-visible sequence after resize and tiling.

Upload bytes and visual tokens belong to different capacity plans. File size affects network transfer, object storage, and image decoding. After decoding, the vision route usually resizes, crops, tiles, or samples the pixel grid before the encoder and connector determine the model-visible sequence.

For a patch-based route, a useful estimate is:

$$N_{patch} = \sum_{i=1}^{V}\left\lceil\frac{H_i}{P}\right\rceil\left\lceil\frac{W_i}{P}\right\rceil, \qquad N_{model} = f_{connector}(N_{patch})$$

Here, $V$ is the number of admitted views or frames, $H_i$ and $W_i$ are their post-preprocessing dimensions, $P$ is the patch size, and $f_{connector}$ describes projection, merging, or queried reduction. Provider APIs may use different tile formulas or fixed budgets, so production estimates must follow the selected route's documented accounting.

A 12 MB JPEG and a 400 KB WebP can produce the same patch count when they decode to the same dimensions and follow the same resize and crop policy. Their transfer and decode costs can still differ, and compression artifacts can change model quality. Conversely, a small compressed file can create a large visual sequence after decoding and tiling.

Admission should therefore enforce at least two separate limits:

• an upload-byte limit for transport, storage, decoder safety, and decompression-bomb protection;
• a projected visual-token limit for encoder work, prefill or cross-attention cost, and memory.

Why can a 500 KB screenshot cost more model compute than a 10 MB photograph?

Answer

The screenshot may produce more post-resize tiles, crops, frames, or patch tokens on the selected vision route. File bytes govern transport and decode concerns; post-preprocessing geometry and the connector govern the model-visible token budget.

Solutions

1. Spatial compression: Pooling or resampling visual tokens (e.g., 256 → 64).
2. Temporal compression: Sampling keyframes or using temporal attention layers (e.g., TimeSformer (Time-Space Transformer)^{[15]Reference 15Is Space-Time Attention All You Need for Video Understanding?https://arxiv.org/abs/2102.05095}, VideoMAE (Video Masked Autoencoders)^{[16]Reference 16VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training.https://arxiv.org/abs/2203.12602}).
3. Dynamic resolution: Tiling or cropping high-resolution inputs so detailed regions get more compute than the rest of the image.

Those three levers decide how many visual tokens fusion will even see. Fusion then decides where the surviving tokens meet text.

Fusion strategies

The visual sequence now exists. Ask where it should meet text: in one shared prefix, through a side memory, or only in a retrieval score. That meeting point controls both the evidence interaction and the serving bill.

With that question in mind, compare where text and visual features meet. Early fusion mixes tokens into one sequence before the transformer. Cross-attention lets configured language layers read visual memory. Late fusion pools each side independently and only merges at the end. None of these placements guarantees grounding; each defines what evidence and serving cost the model can incur.

What question does a fusion strategy answer?

Answer

It decides where modalities meet: before the transformer as one mixed sequence, inside the transformer through cross-attention, or after independent encoders through pooled embeddings.

Early fusion (visual prefix or interleaving)

In early fusion, projected modality tokens are inserted directly into the model input sequence, usually as a contiguous visual prefix or at special placeholder positions inside the prompt. After the projector creates decoder-compatible token states, the model treats image tokens and text tokens as one long sequence and runs standard self-attention over the whole thing. This is used in LLaVA-style architectures and published native multimodal transformers such as Llama 4:

For the copilot's five-view example, predict the admission result before reading the implementation: 2,880 visual tokens plus a 2,000-token transcript already exceed the 3,796-token prefix reserve. The reducer changes that decision, not merely the model's representation.

Pros

Provides token-level image-text interaction through shared self-attention from the first decoder layer that receives the prefix.

Cons

Computationally expensive for long visual sequences, as image tokens consume a large portion of the shared context-window capacity.

When is early fusion a good fit?

Answer

Evaluate early fusion when generation needs token-level interaction between text and image features. Avoid an unbounded prefix when long videos or many high-resolution crops would exceed context or KV-cache budgets.

Late fusion

Late fusion is common in dual-encoder retrieval or classification systems. Each modality is encoded mostly independently, pooled into a compact embedding, and merged by a lightweight head. This design supports indexed scoring, but can't by itself produce answers grounded in detailed visual tokens because a generative decoder is absent from that path:

If the task ends at “which dashboard looks similar?”, that compact score may be enough. If the task asks “which region proves the spike?”, the decoder needs access to surviving visual evidence.

Pros

Highly modular for scoring. Encoders can be swapped independently, and a retrieval path avoids a long generative multimodal prefix.

Cons

Insufficient by itself for grounded generation. Because the scoring path never presents token-level image evidence to a decoder, OCR-heavy answering or region-cited reasoning needs an additional model path.

Why is late fusion insufficient by itself for grounded generation?

Answer

It pools each modality before merging, so this path has no decoder attending over detailed visual tokens. Use it for scored retrieval or classification; route OCR-heavy QA and image-grounded answers through an evidence-aware generative path.

Cross-attention fusion (Flamingo-style)

Cross-attention layers allow text states to compute context-dependent weights over visual features at configured model depths, as seen in Flamingo.^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} Softmax turns the compatibility scores into weights. The gated layer takes the LLM's hidden text state and frozen visual features as inputs, returning a fused representation. Flamingo initializes the gate at zero so training begins from the language-only path and can learn to admit visual information.

The operational question is whether repeated, selective reads of a visual bank cost less than putting that bank in one shared prefix. The answer depends on text length, visual length, number of fusion blocks, and batching, so keep both routes in the benchmark.

Rough attention cost: each fusion block is $O(T_{\text{text}} \cdot T_{\text{vis}})$ in the text-to-vision product (queries over keys), while early-fusion self-attention over a mixed sequence is $O((T_{\text{text}}+T_{\text{vis}})^2)$. Cross-attention can keep visual tokens out of the main prefix KV, but it still pays for every configured fusion layer; early fusion pays quadratic growth in the combined length once, then standard decode on the shared cache.

$$\text{Attention}(Q_{\text{text}}, K_{\text{vis}}, V_{\text{vis}}) = \text{softmax}\left(\frac{Q_{\text{text}} K_{\text{vis}}^T}{\sqrt{d_k}}\right) V_{\text{vis}}$$

Architecture

Gated cross-attention is injected between standard LLM layers to incorporate frozen visual features:

At initialization, a zero gate leaves the language path unchanged by the visual residual. Training can then learn non-zero visual contributions.

Compared with early fusion, this keeps dense visual tokens out of the main decoder prefix. It doesn't make vision free: every configured cross-attention block still performs extra projections and attention against visual memory.

Why does gated cross-attention start with a near-zero gate?

Answer

It lets the model begin as the stable pre-trained text LLM, then gradually learn how much visual information to inject. That reduces training shock from adding a new modality pathway.

Training strategies

With the interface chosen, ask what needs to learn first: the bridge from modality features into the language space, or the whole multimodal stack. That distinction sets the first experiment and its failure diagnosis.

Two-stage training (LLaVA protocol)

Predict the first stage's narrow objective: if the encoders and LLM stay frozen, only the projector can learn to make visual features readable.

The reported LLaVA recipe for visual instruction tuning^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}, later refined in LLaVA-1.5^{[14]Reference 14Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744}:

Stage 1: Feature Alignment

• Freeze: Vision encoder + LLM
• Train: Projection layer only
• Data: ~595K image-text pairs from a filtered CC3M (Conceptual Captions 3M) subset^{[17]Reference 17Conceptual Captions: A Cleaned, Hyperponymed, Image-Caption Dataset for Automatic Image Captioning.https://aclanthology.org/N18-2012/}
• Objective: Predict caption tokens autoregressively through the frozen LLM while training only the projection layer.

Stage 2: Visual Instruction Tuning

• Freeze: Vision encoder (in the reported LLaVA recipe)
• Train: Projection layer + LLM (full fine-tune in the reported recipe; LoRA is a lower-cost implementation choice)
• Data: ~158K GPT-4-generated multimodal instructions across conversations, detailed descriptions, and complex reasoning^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}.
• Objective: Learn to follow instructions and reason about visual content.

Those counts describe the original LLaVA recipe: a filtered CC3M pretraining set plus 158K GPT-4-generated multimodal instructions^{[3]Reference 3Visual Instruction Tuning.https://arxiv.org/abs/2304.08485}. LLaVA-1.5 kept the same two-stage shape, but swapped in a 336px CLIP encoder, an MLP connector, and a larger public task mixture. Its paper reports 558K pretraining pairs plus 665K instruction examples (about 1.2M total) and full training in about one day on a single 8-A100 node^{[14]Reference 14Improved Baselines with Visual Instruction Tuning.https://arxiv.org/abs/2310.03744}.

The two stages connect like this: align the bridge first, then teach grounded assistant behavior on top of that stable interface. For dashboard inspection, compare a connector-alignment stage against direct instruction tuning on held-out grounding slices. Stage ordering outside the reported LLaVA recipe is an experiment, not a guaranteed improvement.

Why does two-stage visual instruction tuning start with feature alignment?

Answer

Stage 1 teaches the projector to map visual features into the LLM's embedding space while the frozen LLM predicts caption tokens. Stage 2 can then focus on following multimodal instructions instead of learning the bridge from scratch.

End-to-end training

After a bridge baseline works, joint training tests whether moving more of the representation improves the target slices enough to pay for added instability.

While two-stage training keeps the pre-trained components largely frozen, end-to-end training updates the modality encoder, projector, and core transformer together. Jointly trained families such as PaLI^{[4]Reference 4PaLI: A Jointly-Scaled Multilingual Language-Image Model.https://arxiv.org/abs/2209.06794}, Gemini^{[5]Reference 5Gemini: A Family of Highly Capable Multimodal Models.https://arxiv.org/abs/2312.11805}, and Llama 4^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/} represent this direction and aim for a more native multimodal representation. Llama 4 warm-starts from a MetaCLIP-based vision encoder that was first aligned against a frozen Llama, then trains the whole stack on interleaved data; in this published recipe, joint training doesn't mean starting every component from random weights^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/}.

This strategy allows gradients to update cross-modal representations jointly. Instead of holding the vision encoder fixed, training can adapt its features to downstream objectives. Whether this improves chart reading, OCR, or grounding depends on data, optimization, and evaluation.

However, joint training exposes more parameters and objectives to optimization. It requires substantial compute and balanced datasets containing text and multimodal examples. Multimodal updates can conflict with language-modeling behavior, so text-only regression measurement remains part of the training contract.

Trade-offs

• Compute: Higher training cost than a projector-only stage because gradients update the encoder and language model as well.
• Stability: Harder to stabilize; carefully tuned learning rates and loss weighting are required to prevent one modality from dominating.
• Quality: Can adapt the encoder to downstream objectives; measure gains against adapter baselines on target slices.

What is the main risk of end-to-end multimodal training?

Answer

Multimodal gradients can destabilize the language model, causing catastrophic forgetting or weaker pure-text behavior. Joint training requires balanced data, careful learning rates, and measurements of both multimodal and text-only behavior.

Post-training for grounded behavior

If grounding is weak, first predict whether the failure comes from missing evidence or poor behavior around evidence. Post-training addresses the second case.

After instruction tuning, a system stack may add a post-training stage focused on response quality, abstention, and safety. Direct Preference Optimization (DPO)^{[18]Reference 18Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} is one option, but it's not a universal third stage for multimodal systems. Depending on the route, teams may use supervised preference data, rejection sampling, RLHF, or task-specific evaluation loops instead.

The important systems point is that post-training can improve how the model explains uncertainty, refuses unsupported claims, and follows system policy. It can't recover information that was already lost upstream. If the vision encoder missed small text or the projector over-compressed the image, no preference method can reconstruct that detail later.

If a multimodal model hallucinates image details, inspect encoder resolution, connector bottleneck, alignment data, and post-training separately. If required evidence never reaches the decoder, preference tuning can't reconstruct it.

Why can't DPO or RLHF fix missing visual detail by itself?

Answer

Preference tuning can improve abstention and answer style, but it can't reconstruct pixels or tokens lost before the decoder. If the encoder missed small text or the projector compressed it away, the signal is gone.

Training decides what the connector and decoder learn. It doesn't decide whether a 2,880-token dashboard prefix fits in GPU memory at 20 QPS. That's a cache, fusion, and token-budget problem.

Serving the evidence path

Training determines what the model can read. Serving determines whether that path meets latency, memory, and concurrency targets when users repeat questions or send large artifacts.

Efficient inference: token budget before parameter budget

Ask which bill dominates before tuning kernels: encoding, visual prefill, cross-attention, or autoregressive decode. Multimodal latency includes modality encoding and processing visual features inside the decoder. In early-fusion systems, that second cost can show up as a large prefill over visual tokens. In cross-attention systems, the prefix stays smaller, but each fusion block still attends over visual memory. Measure these components by route before choosing whether to compress inputs or optimize decode. Visual tokens in the main prefix consume KV cache like text tokens. PagedAttention-style serving makes large, irregular KV caches easier to manage under batching pressure^{[19]Reference 19Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}.

If users ask multiple questions about the same image, caching encoder features avoids rerunning the vision frontend. Caching projected tokens avoids repeating projection. Only a compatible prefix/KV-cache reuse mechanism avoids repeating decoder prefill over an identical visual prefix.

For a follow-up question, predict the stages that can repeat. A cache hit at one boundary saves only work downstream of that boundary; it doesn't make an incompatible representation safe to reuse.

What should you cache for repeated questions about the same image?

Answer

Cache encoder features to avoid rerunning the frontend, or projected visual tokens to avoid projection. To skip repeated decoder prefill for an identical prefix, the serving engine must also reuse compatible prefix/KV-cache state.

Where MoE helps, and where it doesn't

If active feed-forward FLOPs fall but p95 latency and memory stay flat, look outside the experts. The encoder, visual prefix, and batching policy may still dominate.

Mixture-of-Experts (MoE) layers activate only a subset of experts per token, which lowers active FLOPs at a given parameter count^{[20]Reference 20Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961}. That can improve throughput for very large multimodal backbones, but it doesn't remove the main multimodal bottlenecks:

• Vision or audio encoders still have to run.
• In early-fusion systems, long visual prefixes still bloat the KV cache.
• High-resolution inputs still create expensive prefills.

Treat MoE as a model-capacity tool, not as the first fix for high-resolution image serving. Llama 4 is a useful reminder: it uses early fusion and MoE together.^{[6]Reference 6The Llama 4 herd: the beginning of a new era of natively multimodal AI innovationhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/} Sparse experts can lower active feed-forward FLOPs per token. They don't delete the visual prefix from the shared backbone or from the KV cache.

Why doesn't MoE solve visual token explosion?

Answer

MoE reduces active feed-forward FLOPs per token, but it doesn't remove encoder cost, high-resolution prefill, or KV-cache growth from long visual prefixes. Token budgeting still comes first.

Handling variable-length visual tokens

Variable input turns the token budget into a policy decision. A dashboard with one tiny badge and a dashboard with five dense charts shouldn't spend the same visual budget by accident.

MoE doesn't shrink the prefix. If the copilot still encodes every crop and every video frame, you need a selection policy. Keeping every patch token from every view can be computationally prohibitive. In early-fusion paths, self-attention grows quadratically with the shared sequence length. Cross-attention paths avoid that prefix growth, but still pay to read the visual bank. Two strategies control this budget while preserving important details.

Tiling and multi-crop encoding

Start with a global thumbnail, then spend a bounded crop budget on regions chosen by layout, OCR, or task policy. The thumbnail preserves overall layout, while selected crops can preserve fine text, tables, or localized objects. Crop selection is itself an evaluated component: a skipped region can't be recovered downstream.

More crops improve OCR and small-object reasoning, but every crop adds another encoder pass and another batch of visual tokens. That makes this strategy best for documents, charts, maps, and other inputs where fine detail matters.

Newer native multimodal models push this further. Qwen2.5-VL processes images at their native resolution with a dynamic-resolution ViT, then merges each 2×2 patch block into one token so the visual token count tracks actual image size instead of a fixed grid^{[8]Reference 8Qwen2.5-VL Technical Reporthttps://arxiv.org/abs/2502.13923}. The systems goal is the same as tiling: spend visual tokens where detail lives, and keep the budget bounded everywhere else.

When is tiling worth the extra tokens?

Answer

Evaluate tiling when fine details such as small text, labels, charts, maps, or local warning badges matter. A thumbnail can preserve global layout, while selected crops may preserve evidence that downscaling would erase.

Because the number of crops depends on aspect ratio and resolution, the resulting token sequence length is variable. A wide panorama might need a horizontal strip of crops, while a long document might need a vertical stack:

Before encoding, rank candidate regions against the question and predict which crop gets dropped at the limit. That decision is part of model quality, not only preprocessing.

Queried reduction (Perceiver / Q-Former[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597)

If crop count and frame rate vary too much for predictable serving, a fixed query bank offers a different control point. It trades some detail for a bounded downstream sequence.

This approach (popularized by models like Flamingo^{[12]Reference 12Flamingo: a Visual Language Model for Few-Shot Learning.https://arxiv.org/abs/2204.14198} and BLIP-2^{[2]Reference 2BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.https://arxiv.org/abs/2301.12597}) uses a fixed set of learned query vectors to scan the visual features and compress them into a predefined token count. By decoupling the visual input resolution from the sequence length passed to the language model, this architecture helps prevent out-of-memory failures and keeps the effective visual token budget bounded even for high-framerate video or dense spatial layouts.

The fixed-query pattern in Q-Former and Perceiver Resampler designs cross-attends to a potentially large sequence of visual features. It converts variable-length visual input into a configured fixed-size token budget, whose retained evidence still needs task-specific evaluation.

What tradeoff does queried reduction make?

Answer

It converts variable-length visual features into a fixed token budget, improving serving predictability. Compression can discard fine detail if the query count or training data is insufficient.

Audio integration

Treat the bridge clip as a timeline, not as another image. The useful question is often “what was said before the spike?” so the representation must preserve enough temporal detail to attach words or sounds to evidence.

The copilot's third frontend is the incident bridge. The same encoder-connector-LLM pattern applies, but the budget problem is time, not patches. A 40-second clip at 100 Hz is 4,000 frontend steps before any fusion.

Speech-heavy pipelines often start from log-Mel spectrograms, which is the path Whisper uses^{[10]Reference 10Whisper: Robust Speech Recognition via Large-Scale Weak Supervision.https://arxiv.org/abs/2212.04356}. Other audio encoders learn a waveform frontend directly instead of hand-specifying a spectrogram. For data-center alarms, fan noise, relay clicks, or other non-speech signals, contrastive models like CLAP (Contrastive Language-Audio Pretraining)^{[11]Reference 11CLAP: Learning Audio Concepts from Natural Language Supervision.https://arxiv.org/abs/2206.04769} are more appropriate.

Once audio is encoded into feature vectors, a connector can map those representations into prefix-token space or expose them as separate memory for cross-attention. Temporal compression (strided convolutions or pooling) cuts sequence length before downstream fusion:

Predict the serving effect before changing stride: stride 10 cuts 4,000 frontend steps to 400, but a short alarm or timestamped phrase may be harder to localize. Measure temporal attribution alongside latency.

How does audio fit the same multimodal architecture pattern?

Answer

Audio uses an audio frontend or encoder, then a connector exposes the resulting time-frequency or learned features through a projected prefix or cross-attention memory. Long audio needs temporal compression before either path grows too expensive.

Practical considerations

When an answer fails, trace the evidence boundary in order: was the artifact admitted, did the frontend preserve the detail, did the connector expose it, did fusion read it, and did the evidence gate validate the citation? This keeps decoder fluency from hiding an upstream miss.

Serving already covered encoder pools, visual prefill, and KV growth. The remaining production failures are optimization instability, image-borne prompt injection, and using extra decode steps as a substitute for perception.

Training instability

Suppose text-only quality drops after a connector or encoder is unfrozen. Treat that regression as a signal to compare modality-specific learning rates, gradients, data balance, and grounded slices, not as proof that one component is solely responsible.

Balancing optimization across modalities during end-to-end or partially unfrozen training can be difficult. There often isn't a clean "vision loss" fighting a separate "text loss." Multimodal batches, connector updates, or modality-specific learning rates can drag the language model away from its pre-trained distribution, showing up as weaker pure-text behavior.

A possible contributor is mismatch between image features and decoder input states early in training, often called the modality gap. If the connector isn't aligned adequately, grounding quality can suffer. Track learning rates, gradients, pure-text regressions, and grounded task slices rather than assuming one symptom has one cause.

What is the modality gap?

Answer

It's a mismatch between modality features and the representation interface the language model learned to consume. A poorly aligned connector can cause weak grounding; verify with grounded task slices and ablations.

Safety & adversarial attacks

Suppose OCR reads “ignore previous instructions” from a screenshot. That string came from an artifact, not from the operator's intent, so the system needs a trust boundary before it reaches action or policy logic.

Visual inputs introduce attack vectors that text-only guardrails can't catch. A screenshot, PDF, or image can contain prompt injection text that the OCR or vision stack faithfully reads. The model isn't "cheating" here. It's doing exactly what it was trained to do: extract and follow visible instructions.

Adversarial or misleading visuals can also change model behavior even when raw pixels look benign to a human viewer. A production threat model should cover image moderation where needed, OCR-derived instruction isolation, and downstream policy checks on final actions or answers.

Why do visual inputs create prompt-injection risk?

Answer

Screenshots, PDFs, and images can contain text instructions that OCR or the vision model reads. The assistant may treat those visible instructions as user intent unless the pipeline sanitizes image-derived text and applies downstream policy checks.

Reasoning after perception

Giving the decoder more reasoning budget can help on charts, diagrams, and long documents, but it's orthogonal to multimodal architecture. Extra compute only helps after perception succeeds. If the visual encoder missed a label or the projector discarded detail, more decoding steps won't bring that signal back.

When will extra reasoning budget fail to improve multimodal answers?

Answer

When the perception stage lost the relevant evidence. More decoder steps can reason over captured labels, chart marks, or objects, but can't recover detail the encoder never saw or the projector discarded.

Grounding evaluation should reflect that boundary: a fluent answer doesn't pass when the cited region or OCR evidence is absent.

Read the release gate as a diagnosis, not a single score. If helpfulness clears its bar while OCR exact match misses, improve perception or the connector before tuning answer style.

Common pitfalls

When a grounded answer is wrong, use the trace to locate the first lost dependency. The pitfalls below name the tempting shortcut at each boundary and the experiment that can falsify it.

"Multimodal design is mostly token concatenation"

• Symptom: The system can ingest image or audio tokens, but grounded answers stay shallow or brittle.
• Cause: Encoder, connector, and fusion choices were treated as wiring details instead of the real architecture.
• Fix: Design the full path explicitly: which encoder preserves evidence, how the connector compresses or exposes it, and how the LLM attends back to it.

"Hallucinated detail means the LLM needs more tuning"

• Symptom: The model invents labels, counts, or chart details even though the training data is clean.
• Cause: Evidence was lost earlier in the encoder or connector, so decoding is guessing from a weak representation.
• Fix: Debug in order: encoder resolution, connector bottleneck, alignment data, then post-training.

"Late fusion is fine for grounded generation"

• Symptom: The system scores images and text well but fails when it has to explain or point to supporting evidence.
• Cause: Late fusion keeps modalities too separate for token-level grounded generation.
• Fix: Use early fusion or cross-attention when the answer must stay tied to visual or audio evidence during decoding.

"Token budget only matters for text"

• Symptom: Prefill latency and KV-cache growth explode after adding high-resolution images or long audio clips.
• Cause: Visual and audio prefixes were treated as free context before the LLM.
• Fix: Budget multimodal tokens first. Use tiling, queried reduction, sparse sampling, or temporal compression before expanding the backbone.

"Preference tuning can recover missing perception"

• Symptom: Teams keep tuning policy behavior while OCR, chart reading, or grounding accuracy stays flat.
• Cause: Training is targeting the decoder after the evidence path has already failed.
• Fix: Fix encoder and connector quality first, then use post-training for abstention, policy alignment, or output style.

"MoE solves high-resolution multimodal cost by itself"

• Symptom: Backbone FLOPs drop, but multimodal serving is still slow and memory-heavy.
• Cause: MoE only sparsifies part of the backbone; it doesn't reduce encoder cost or long multimodal prefixes.
• Fix: Treat MoE as a backbone-capacity tool. Handle image, video, and audio token budgets separately.

Common design scenarios

Two system design scenarios show up frequently in AI engineering work:

Scenario: Design real-time video QA. How do you prevent continuous frame embeddings from overflowing the KV cache?

Answer

Avoid frame-by-frame early fusion. Use sparse frame sampling plus temporal compression, such as a Perceiver-style reducer, to distill clips into a fixed token budget. At serving time, use an eviction or sliding-window policy that preserves compressed state while dropping stale frame tokens.

Scenario: A VLM fails on 8K dashboard screenshots because raw ViT embeddings exceed memory. What design should you use?

Answer

Use dynamic tiling plus a global thumbnail. The thumbnail keeps overall layout, while tiles preserve axis labels, warning badges, small legends, and other fine details. If memory is tight, encode tiles sequentially, then concatenate or resample the visual tokens before LLM prefill.

Multimodal architecture checklist

Use this checklist at design review after walking one artifact through the evidence path. Each item should map to a measurable capacity, quality, or safety decision.

1. The core decomposition is Modality Encoder -> Connector -> Language Backbone.
2. Connector choice determines how much visual detail survives and whether evidence becomes a projected prefix or a separate memory bank. Simple MLPs work well for direct projection, while Q-Former and Perceiver-style reducers keep downstream visual counts bounded.
3. Early fusion is common for generative VLMs. Cross-attention is useful when you want to keep visual features separate. Late fusion fits retrieval and classification more than grounded generation.
4. Two-stage alignment plus instruction tuning is a documented adapter-stack recipe. Published or disclosed native multimodal models train more of the multimodal path together. Compare task quality, compute, and regressions; preference tuning is optional, not universal.
5. The biggest serving bottlenecks are encoder latency, visual-feature prefill cost, and, in early-fusion designs, KV cache growth. Cache visual features and manage memory carefully.
6. MoE can reduce active FLOPs, but it doesn't solve visual token explosion by itself. Llama 4 uses early fusion and MoE together; the visual prefix still occupies the shared backbone.

Treating multimodal input as just "text with extra tokens" is the recurring failure mode. Concatenation is easy compared with deciding what to encode, what to compress, and what the LLM should attend to. A model that reads a Grafana panel still isn't a model that can emit one.