Diffusion Models: Images & Text

Four previews of "oak dining table in soft light" arrive as one HTTP request. Producing them isn't one piece of work: each image starts as random noise and passes through a network repeatedly before anyone can see it. The designer wants a preview in under two seconds. How much of that time can the repeated network calls consume?

That repeated repair loop is diffusion. The network is a denoiser: it predicts information a numerical update uses to remove noise. Image models operate on pixels or compressed image representations called latents. Text models can apply a related idea to a discrete token canvas, refining uncertain positions instead of appending one token at a time.^{[1]Reference 1DiffusionGemmahttps://ai.google.dev/gemma/docs/diffusiongemma}

The VAE lesson introduced learned image compression; attention showed how queries retrieve information from keys and values. Here those pieces become a prompt-conditioned generation loop. Keep the service requirements in view, then work through one brightness value before scaling to full images.

Product brief: a governed design-asset generator

Design a service for product teams that creates synthetic campaign backgrounds and illustration previews. First decide where a slow or unsafe job can stop: before GPU work, inside the denoising loop, or after pixels are decoded. It must never present generated UI text, charts, or incident screenshots as verified evidence.

Requirements and API

• POST /v1/image-jobs accepts prompt, negative_prompt, width, height, variant_count, route (preview or quality), seed, and an idempotency key. It returns job_id, accepted model and policy versions, and an estimated completion window.
• GET /v1/image-jobs/{job_id} returns queued, running, review, failed, or complete, plus per-variant status, seed, sampler, step count, safety disposition, watermark result, and synthetic-content label. Asset URLs appear only for variants that passed release checks.
• Preview requests target p95 first released preview below 2 seconds, measured from admission, including queueing and checks. Quality jobs must reach complete, review, or failed within 30 seconds; human review has its own deadline.
• Prompts and outputs pass policy checks. Logo use, realistic UI claims, and safety-sensitive imagery route to review. Tenant data and fine-tune adapters stay isolated.

Data flow and sizing

Store model, adapter, scheduler, policy, and per-variant seeds with every asset so an operator can explain its generation. Size the GPU loop in per-image denoiser predictions, not HTTP requests or Python function calls. A peak of 20 preview jobs/s with 4 variants and a 6-step distilled route needs 20 x 4 x 6 = 480 predictions/s. This is offered work, not a throughput claim for a GPU.

A smaller quality lane uses classifier-free guidance (CFG), which compares prompt-conditioned and unconditioned predictions at each step. At 2 jobs/s with 4 variants, 50 steps, and ordinary CFG, that lane needs 2 x 4 x 50 x 2 = 800 predictions/s. A batched network call can compute several predictions; it doesn't erase this work. Compare capacities only at the same model, resolution, precision, and batching policy.

Use bounded queues and reserve worker capacity for previews. Separate queue names alone don't prevent starvation on shared GPUs. Reject excess work before its deadline becomes impossible; cap dimensions, variants, and steps before admission. Batch only compatible model/adapter versions, shapes, schedules, and guidance modes. A first-preview priority may return one released variant while siblings are still running.

Admission rejects invalid or disallowed work before GPU spend. Prompt embeddings and seeded noise are separate inputs to generation: the text encoder doesn't create the random latent. The VAE decoder turns the final compressed state into pixels. Release can still withhold a visually convincing result.

Provenance belongs inside the asynchronous release path

A synthetic label in the API response is easy to lose after the file leaves the product. For this service, choose a watermark-capable generation route, store a separate provenance record, and verify both before publication. Google DeepMind's SynthID is one example of an imperceptible digital watermark added at creation, with robustness against common edits such as cropping and lossy compression.^{[2]Reference 2SynthIDhttps://deepmind.google/models/synthid/} This reference doesn't promise a general-purpose SynthID image-embedding API for an arbitrary self-hosted model.

That is a robustness target, not a cryptographic guarantee that survives every edit. Detection may be uncertain, and a missing watermark doesn't prove that the media is human-made. Treat the watermark as one provenance signal alongside content policy, metadata, and human review.

The job record should bind job_id, asset digest, model and adapter versions, seed, sampler, policy version, watermark method and version, detector version, detector result, and publication time. Keep the pre-watermark render private.

Watermark placement depends on the chosen implementation: it may happen inside generation or after decode. Prepare permitted sizes and encodings privately, then check the exact derivative that will be distributed. Disable unchecked CDN transforms; otherwise a later resize can invalidate evidence about the published bytes.

Publish an immutable object only after its digest and manifest commit. An object left behind by a failed database transaction stays private for later cleanup. Repeated release attempts reuse the committed bytes rather than overwriting an asset under the same URL.

Measure watermark and detector latency separately from denoiser evaluations. A fast GPU sampler won't hide a slow release path.

Failure recovery, rollout, and evaluation

Bind an idempotency key to the authenticated tenant and normalized request hash in durable storage. An identical retry returns the same job; the same key with a different prompt, shape, or route returns a conflict. Authorize every status request and asset read against the tenant, not possession of a job ID.

Queue delivery may duplicate. Give each worker attempt an expiring lease and increasing attempt number. Completion is a conditional update against the current attempt, so a timed-out worker can't publish after its replacement. Pin each variant's seed and immutable configuration before dispatch; changing batch order mustn't silently assign it a sibling's random stream.

A stochastic denoising diffusion probabilistic model (DDPM) retry must restore the random-number-generator state or restart the full sample. Resuming with fresh noise changes the asset. Timeout, out-of-memory, or failed safety checks move the job to failed or review rather than publishing a partial image.

A seed controls random draws, not every source of numerical nondeterminism. Record the execution environment, precision, batching, VAE, and library versions; hardware or kernel changes can alter results. Mid-sample recovery also needs the latent, timestep, and scheduler history. Reuse stored bytes when exact identity matters instead of promising that a rerun will recreate them.^{[3]Reference 3Reproducibilityhttps://huggingface.co/docs/diffusers/using-diffusers/reusing_seeds}

Roll out a new model or sampler against a frozen prompt suite, then shadow it, canary 1% of eligible jobs, expand to 10%, and finally ramp by route. Keep the candidate and baseline on the same fixture set so a change in prompt mix doesn't masquerade as a model improvement.

Roll back on safety-policy regressions, memorization alerts, p95 latency breaches, or a material drop in human preference. Evaluation covers prompt adherence, human preference, diversity, text and layout fidelity, unsafe-output rate, memorization checks, p95 latency, failure rate, and cost per accepted asset.

A one-pixel forward process first

Before looking at a scheduler or a denoiser, answer one small question: at $\bar{\alpha}=0.5$, how much of a clean pixel remains? Fix one normalized pixel with brightness $x_0 = 1.0$ and a known noise sample $\epsilon = -0.4$. At each noise level $\bar{\alpha}$, the closed-form mix is:

$x_t = \sqrt{\bar{\alpha}} \, x_0 + \sqrt{1 - \bar{\alpha}} \, \epsilon$

At $\bar{\alpha}=0.5$, signal amplitude is $\sqrt{0.5}\,x_0 \approx 0.707\,x_0$, not half brightness. The schedule's SNR is a ratio of squared coefficients, assuming unit-variance signal and noise. It's 1.0 here, not the ratio of this particular pixel's squared signal and sampled noise.

At $\bar{\alpha}=0.001$, the noise dominates. We reuse one noise draw to compare coefficients, not to simulate a Markov trajectory. Independent training rows draw their own noise. Before running the calculation, check the middle row by hand:

import json
import math

def forward_diffusion_value(x0: float, alpha_bar: float, epsilon: float) -> dict[str, float]:
    if not 0 <= alpha_bar < 1:
        raise ValueError("alpha_bar must be in [0, 1); SNR diverges at 1")
    signal = math.sqrt(alpha_bar) * x0
    noise = math.sqrt(1 - alpha_bar) * epsilon
    return {
        "alpha_bar": alpha_bar,
        "signal": round(signal, 3),
        "noise": round(noise, 3),
        "xt": round(signal + noise, 3),
        "snr": round(alpha_bar / (1 - alpha_bar), 3),
    }

x0 = 1.0
epsilon = -0.4
steps = [
    forward_diffusion_value(x0, alpha_bar, epsilon)
    for alpha_bar in [0.9, 0.5, 0.001]
]

print(json.dumps(steps, indent=2))

[
  {
    "alpha_bar": 0.9,
    "signal": 0.949,
    "noise": -0.126,
    "xt": 0.822,
    "snr": 9.0
  },
  {
    "alpha_bar": 0.5,
    "signal": 0.707,
    "noise": -0.283,
    "xt": 0.424,
    "snr": 1.0
  },
  {
    "alpha_bar": 0.001,
    "signal": 0.032,
    "noise": -0.4,
    "xt": -0.368,
    "snr": 0.001
  }
]

Apply the same formula elementwise to every pixel or latent channel. Generation later reverses the path: it starts near pure noise and cleans structure back in step by step.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239}

Why learn many cleanup tasks?

A one-shot mapping from a random vector to a 512x512 background sounds attractive. Generative Adversarial Networks (GANs) pair that generator with a discriminator. Their adversarial objective can be difficult to tune and can suffer mode collapse, while strong GAN implementations can still produce excellent samples. Diffusion changes the training objective and serving cost, not a guarantee of better campaign art.

The ordinary diffusion route doesn't ask the network to generate an image in one pass. It predicts noise or another denoising target at the current noise level; a scheduler turns that prediction into the next state. Repetition turns a random field into a structured scene. A distilled route can learn larger jumps, but it still has to be trained for that job.

Classical DDPM training starts from an evidence lower bound (ELBO), a trainable lower bound on data log-likelihood. Its common simple objective is a noise-prediction regression loss derived from that bound.

Unlike GANs, this removes the adversarial minimax game. It doesn't by itself prove coverage, memorization safety, or output suitability.

Count work before declaring a winner. GANs, Variational Autoencoders (VAEs), and diffusion models trade off sample quality, coverage, likelihood modeling, and inference cost. For campaign backgrounds, the costly column is inference work: one generator pass versus 6 preview steps or 50 quality steps. Treat the table as an architecture comparison, not a universal ranking:

How the oak table becomes noise

Adding noise gradually

Ask what training must show the denoiser. It takes a clean campaign still, turns the table and room into isotropic noise, and asks the model to learn the reverse. The one-pixel table above is that story for one brightness value. The formal process extends it to full tensors and draws its principles from non-equilibrium thermodynamics.^{[5]Reference 5Deep Unsupervised Learning using Nonequilibrium Thermodynamics.https://arxiv.org/abs/1503.03585}

Starting from a clean image $x_0$, we add Gaussian noise over $T$ steps. This fixed Markov chain means each new state depends only on the previous one, not on the full history. With enough cumulative noise, the terminal distribution approaches isotropic Gaussian noise: independent coordinates with the same variance on every pixel and color channel.^{[5]Reference 5Deep Unsupervised Learning using Nonequilibrium Thermodynamics.https://arxiv.org/abs/1503.03585}

The transition kernel generalizes the one-pixel mix. Each step preserves a little signal and adds a little Gaussian noise:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} \cdot x_{t-1}, \beta_t \mathbf{I})$

Here $\beta_t$ is the noise schedule. The original DDPM experiments used a linear schedule from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$ over $T = 1000$ steps; newer systems may choose different schedules.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239} Because sums of Gaussians remain Gaussian, the derivation can marginalize over intermediate steps. Let $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} \cdot x_0, (1 - \bar{\alpha}_t) \mathbf{I})$

This is the formula you computed by hand. Sample $x_t$ at any timestep directly from $x_0$ and one noise draw without simulating intermediate states.

For the cited 1000-step schedule, $\bar{\alpha}_T \approx 0.0000404$, so the terminal distribution is close to $\mathcal{N}(0, \mathbf{I})$. A short or weak schedule needn't reach this regime; check its cumulative product before assuming a Gaussian starting distribution will match training.

At a small forward timestep $t$, the input retains recognizable structure. At a large forward $t$ near $T$, it retains very little original signal. The SNR curve $\bar{\alpha}_t / (1 - \bar{\alpha}_t)$ describes that spectrum. Keep its direction separate from reverse sampling: forward time adds noise, while reverse time removes it.

Reverse process: learned denoising

DDPM training

Training has one job: approximate the reverse process $p_\theta(x_{t-1} | x_t)$, the step-by-step cleanup that generation will run later.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239}

Why not compute that reverse step exactly? The posterior $q(x_{t-1} \mid x_t)$ marginalizes over every clean image $x_0$ that could have produced $x_t$, and that marginalization makes it intractable.

The conditional posterior $q(x_{t-1} \mid x_t, x_0)$ is tractable and remains Gaussian. DDPM uses that closed form in its derivation, then trains a model to predict enough information from $x_t$ and $t$ to approximate the reverse step without seeing $x_0$ at inference time.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239}

For $t>1$, its mean and variance are:

See why the extra condition matters with two equally likely clean values, $x_0=-1$ or $+1$. Choose $\bar{\alpha}_{t-1}=0.5$, $\beta_t=0.5$, hence $\bar{\alpha}_t=0.25$, and observe $x_t=0$. Both clean values are equally plausible by symmetry:

If $x_0$ is unknown, the reverse distribution here is an equal mixture of those two Gaussians, not either single Gaussian. Its mean is zero, but its variance is $\frac13+(\frac{\sqrt2}{3})^2=\frac59\approx0.556$, including uncertainty about the clean value. This tiny distribution can be calculated; integrating over a real image distribution is the hard part. DDPM's learned Gaussian transition is an approximation to that marginal reverse step.

The model doesn't have to predict the clean image $x_0$ directly. In standard DDPM (Denoising Diffusion Probabilistic Models), it predicts the noise component $\epsilon$ added at step $t$.

Train a neural network $\epsilon_\theta(x_t, t)$ to recover that noise from a noisy input:

$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$

For each row, sample a timestep $t$, a clean image $x_0$, and noise $\epsilon$. Build $x_t$, ask the network to predict $\epsilon$, and average the squared error over many rows.

In practice, the "simple" objective is an MSE loss on predicted noise. It comes from simplifying and reweighting the original ELBO, but the implementation is supervised regression from $x_t$ to $\epsilon$.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239} The small program below makes that target explicit:

import json

def mse(true_values: list[float], predicted_values: list[float]) -> float:
    squared_errors = [
        (true - predicted) ** 2
        for true, predicted in zip(true_values, predicted_values, strict=True)
    ]
    return sum(squared_errors) / len(squared_errors)

training_rows = [
    {
        "timestep": 100,
        "true_noise": [0.20, -0.10, 0.05],
        "predicted_noise": [0.18, -0.08, 0.02],
    },
    {
        "timestep": 700,
        "true_noise": [0.90, -0.40, 0.30],
        "predicted_noise": [0.72, -0.35, 0.10],
    },
]

losses = [
    {
        "timestep": row["timestep"],
        "mse_loss": round(mse(row["true_noise"], row["predicted_noise"]), 4),
    }
    for row in training_rows
]

print(json.dumps(losses, indent=2))

[
  {
    "timestep": 100,
    "mse_loss": 0.0006
  },
  {
    "timestep": 700,
    "mse_loss": 0.025
  }
]

These are fixed predictions for inspecting the loss, not a training run. The second row has larger error; two hand-chosen rows don't establish that larger timesteps are always harder. Across training, the same network must handle inputs from nearly clean images to nearly pure static.

There is no adversarial game and no discriminator. The ground-truth target is available because training knows exactly which $\epsilon$ it injected.

Noise schedule and SNR

In the closed-form forward equation, the signal coefficient is $\sqrt{\bar{\alpha}_t}$ and the noise coefficient is $\sqrt{1 - \bar{\alpha}_t}$, so the effective signal-to-noise ratio at timestep $t$ is:

$\mathrm{SNR}(t) = \frac{\bar{\alpha}_t}{1 - \bar{\alpha}_t}$

High-SNR timesteps still contain recognizable structure. Low-SNR timesteps are close to pure noise.

If the schedule destroys signal too quickly, many late timesteps become almost uninformative and the model spends capacity on inputs with little semantic content left. Schedule design and loss weighting therefore decide where the model spends its learning budget along the SNR curve.

Prediction targets

Original DDPM predicts $\epsilon$ directly.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239} That isn't the only valid parameterization. Some systems predict the clean sample $x_0$; others use $v = \sqrt{\bar{\alpha}}\,\epsilon - \sqrt{1-\bar{\alpha}}\,x_0$, the velocity parameterization used in progressive distillation.^{[6]Reference 6Progressive Distillation for Fast Sampling of Diffusion Models.https://arxiv.org/abs/2202.00512} This $v$ isn't the straight-path flow-matching target introduced later.

These targets are convertible at nondegenerate noise levels, but they change optimization and how errors spread across timesteps. At $a=\sqrt{\bar{\alpha}}$ and $b=\sqrt{1-\bar{\alpha}}$, the inverse is $\hat{x}_0=a x_t-b\hat{v}$ and $\hat{\epsilon}=b x_t+a\hat{v}$. A scheduler configured for epsilon mustn't interpret an unconverted velocity prediction as noise.

Sampling (reverse process)

To generate a new image, start from pure noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ and iteratively denoise it. A DDPM reverse step combines a learned mean with fresh stochastic noise on non-final steps:

Set $z=0$ for the final step into $x_0$. The original formulation allows a chosen reverse variance; the scalar teaching example uses $\sigma_t=\sqrt{\beta_t}$.

Each step depends on the previous state, even when several images share a batched call. The following scalar trace isolates the update arithmetic. Its three betas leave $\bar{\alpha}_T=0.912576$, nowhere near pure noise, and predicted_noise is a fixed function, not a learned model. Starting at x=1.25 exercises the formula; it doesn't demonstrate image generation.

import json
import math
import random

betas = [0.02, 0.03, 0.04]
alphas = [1 - beta for beta in betas]
alpha_bars: list[float] = []
running = 1.0
for alpha in alphas:
    running *= alpha
    alpha_bars.append(running)

def predicted_noise(x: float, step: int) -> float:
    return 0.45 * x + 0.02 * step

x = 1.25
trace = []
rng = random.Random(7)
for step in reversed(range(len(betas))):
    beta_t = betas[step]
    alpha_t = alphas[step]
    alpha_bar_t = alpha_bars[step]
    eps = predicted_noise(x, step)
    mean = (x - (beta_t / math.sqrt(1 - alpha_bar_t)) * eps) / math.sqrt(alpha_t)
    z = rng.gauss(0, 1) if step > 0 else 0.0
    x = mean + math.sqrt(beta_t) * z
    if step == 2:
        checkpoint = (x, step - 1, rng.getstate())
    trace.append({
        "step": step,
        "predicted_noise": round(eps, 3),
        "reverse_mean": round(mean, 3),
        "stochastic_z": round(z, 3),
        "next_x": round(x, 3),
    })

# Resume after step 2, using both the latent and the consumed RNG state.
def resume(saved, generator):
    value, next_step, _ = saved
    for step in reversed(range(next_step + 1)):
        eps = predicted_noise(value, step)
        mean = (value - betas[step] / math.sqrt(1 - alpha_bars[step]) * eps) / math.sqrt(alphas[step])
        z = generator.gauss(0, 1) if step > 0 else 0.0
        value = mean + math.sqrt(betas[step]) * z
    return value

restored_rng = random.Random()
restored_rng.setstate(checkpoint[2])
assert math.isclose(resume(checkpoint, restored_rng), x, abs_tol=1e-12)
assert not math.isclose(resume(checkpoint, random.Random(7)), x, abs_tol=1e-12)
print(json.dumps(trace, indent=2))

[
  {
    "step": 2,
    "predicted_noise": 0.603,
    "reverse_mean": 1.193,
    "stochastic_z": -0.256,
    "next_x": 1.141
  },
  {
    "step": 1,
    "predicted_noise": 0.534,
    "reverse_mean": 1.086,
    "stochastic_z": 0.511,
    "next_x": 1.174
  },
  {
    "step": 0,
    "predicted_noise": 0.528,
    "reverse_mean": 1.111,
    "stochastic_z": 0.0,
    "next_x": 1.111
  }
]

Read the trace as a dependency chain: each new next_x becomes the next step's input, while stochastic_z changes intermediate states. The final step sets it to zero.

The checkpoint follows step 2. Its next draw must be 0.511, not the first seeded draw -0.256. Reseeding with 7 while retaining the saved latent repeats that earlier draw and changes the result; restoring getstate() reproduces the uninterrupted scalar sample. This simple scheduler has no multistep history. A production checkpoint must also retain any such history and match the model, sampler, and execution configuration.

Try setting every z to zero. You remove randomness but still use the DDPM mean formula; this doesn't turn the program into DDIM. The deterministic DDIM update below has different coefficients.

The original DDPM sampler evaluates the denoiser across its configured reverse chain, commonly presented with $T=1000$ steps. That baseline is expensive compared with one-pass generation, which motivates sparse samplers and distilled models. Before taking those shortcuts, we need a denoiser that can predict the noise and a way to inject the prompt into every reverse step.

Model architectures: U-Net and diffusion transformers

The U-Net backbone

A denoiser must recover both layout and small spatial details. For a UI illustration scene, it needs to place panels and visual anchors coherently while preserving edges, textures, and lighting cues. Ask what happens if it builds layout but loses local edges: the scene may be plausible while every label and border is soft.

A U-Net (U-shaped convolutional network with skip connections) addresses that failure with two paths. Its down path builds broader context, its up path restores spatial resolution, and skip connections carry local cues around the deepest bottleneck.

In text-conditioned image models, timestep embeddings modulate residual blocks, and cross-attention is inserted at several resolutions so prompt tokens can steer both coarse layout and fine detail.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}

The drawing compresses the number of blocks and omits channels. Timestep modulation and text conditioning aren't drawn: Stable Diffusion-style U-Nets use cross-attention in multiple down, middle, and up blocks rather than only at the bottleneck.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}

The transformer shift

A Diffusion Transformer (DiT) changes the denoiser's representation. It patchifies the input into tokens and processes them with transformer blocks instead of a convolutional U-Net.

In the original DiT experiments, increasing model compute improved Fréchet Inception Distance (FID) on class-conditioned ImageNet generation.^{[8]Reference 8Scalable Diffusion Models with Transformers.https://arxiv.org/abs/2212.09748} FID compares generated and reference image-feature distributions; lower is better for this metric, but it doesn't directly measure campaign-prompt adherence or safety. Those results support a scaling path, not a claim that every DiT is cheaper or better for this product.

Stable Diffusion 3's MM-DiT (multimodal diffusion transformer) is one published example. Its text and image representations participate in a joint attention operation while retaining separate modality-specific weights, rather than feeding text only through one-directional cross-attention.^{[9]Reference 9Scaling Rectified Flow Transformers for High-Resolution Image Synthesishttps://arxiv.org/abs/2403.03206}

The SD3 authors report prompt-following, typography, and scaling results for that recipe. Those measured results shouldn't be generalized to every transformer-based image generator.

Timestep conditioning

Both backbones need to know how much noise remains. U-Net-style models usually encode the timestep $t$ with a sinusoidal embedding, project it with an MLP, and use the result to modulate residual blocks through FiLM (Feature-wise Linear Modulation) or AdaGN (Adaptive Group Normalization).

DiT carries the same timestep signal, but typically injects it through adaptive LayerNorm inside transformer blocks instead of GroupNorm-based residual blocks.^{[8]Reference 8Scalable Diffusion Models with Transformers.https://arxiv.org/abs/2212.09748} A framework-free scale-and-shift sketch makes the shared idea visible:

import json

def modulate(features: list[float], scale: float, shift: float) -> list[float]:
    return [round(value * (1 + scale) + shift, 3) for value in features]

feature_map = [0.20, -0.10, 0.50]
timestep_controls = {
    "reverse_start_high_noise": {"scale": 0.50, "shift": -0.05},
    "reverse_end_low_noise": {"scale": 0.05, "shift": 0.02},
}

outputs = {
    name: modulate(feature_map, **control)
    for name, control in timestep_controls.items()
}

print(json.dumps(outputs, indent=2))

{
  "reverse_start_high_noise": [
    0.25,
    -0.2,
    0.7
  ],
  "reverse_end_low_noise": [
    0.23,
    -0.085,
    0.545
  ]
}

The controls here are hand-chosen to show scale and shift. A trained model learns them from the timestep embedding; it needn't use larger scale values at higher noise. Broad structure often emerges before fine detail along reverse sampling, but all network blocks run at each noise level.

In the forward noising direction, those same endpoints are called large or late $t$ and small or early $t$, respectively. Name the direction when discussing "early" so the step is unambiguous.

Flow matching and rectified flow

DDPM frames generation as reversing a stochastic noising chain. Flow matching reframes it as learning a velocity field that transports samples along a path between a noise distribution and the data distribution.^{[10]Reference 10Flow Matching for Generative Modelinghttps://arxiv.org/abs/2210.02747}

For a straight conditional path, pair a data sample $x_0$ with independent noise $x_1 \sim \mathcal{N}(0, \mathbf{I})$ and interpolate $x_\tau = (1 - \tau) x_0 + \tau x_1$ for $\tau \in [0, 1]$.^{[11]Reference 11Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowhttps://arxiv.org/abs/2209.03003} Time increases toward noise in this notation, so generation integrates backward from 1 to 0. The regression target for each pair is its constant displacement $x_1-x_0$:

$\mathcal{L}_{\text{FM}} = \mathbb{E}_{\tau, x_0, x_1}\big[\,\|\,v_\theta(x_\tau, \tau) - (x_1 - x_0)\,\|^2\,\big]$

The network sees $x_\tau$ and $\tau$, not both endpoints. Different pairs can cross at the same state, so squared-error training learns their conditional mean velocity. That field generally varies with position and time; its generated trajectories needn't be straight. Rectification can improve their straightness. Straight conditional training paths alone don't guarantee one-step generation.^{[11]Reference 11Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowhttps://arxiv.org/abs/2209.03003[10]Reference 10Flow Matching for Generative Modelinghttps://arxiv.org/abs/2210.02747}

Check the sign with $x_0=1$ and $x_1=-1$: the target is $-2$. From noise at $\tau=1$, an idealized Euler step of $\Delta\tau=-0.25$ moves $x$ from $-1$ to $-0.5$. A positive time step would move away from the data. Stable Diffusion 3 reports rectified flow with MM-DiT and a nonuniform timestep-sampling strategy.^{[9]Reference 9Scaling Rectified Flow Transformers for High-Resolution Image Synthesishttps://arxiv.org/abs/2403.03206}

Text diffusion and DiffusionGemma

Image diffusion corrupts continuous pixel or latent values with Gaussian noise. Text is different: a sentence is a sequence of discrete token IDs. A text diffuser therefore uses a discrete corruption process, often replacing tokens with masks or sampling from categorical transitions, then trains a model to reconstruct original tokens from noisy context.^{[12]Reference 12Structured Denoising Diffusion Models in Discrete State-Spaceshttps://arxiv.org/abs/2107.03006}

Autoregressive language models generate left to right: $p(x_i \mid x_{<i})$. Once token 12 is emitted, token 4 usually won't be revised.

A diffusion language model starts from a noisy or masked sequence and refines many positions over several rounds. Each round can use context from both sides of a missing token, so the model can repair an earlier word after seeing later words.

Google's experimental open-weights DiffusionGemma provides a concrete example, checked October 10, 2026. It builds on the Gemma 4 26B-A4B mixture-of-experts family. The model card reports 25.2B total parameters and 3.8B active, with 8 of 128 routed experts plus one shared expert.^{[13]Reference 13DiffusionGemma Model Cardhttps://ai.google.dev/gemma/docs/diffusiongemma/model_card}

It accepts text, image, and video inputs and generates text with an encoder-decoder stack. An autoregressive encoder prefills the prompt into a KV cache. A decoder then applies bidirectional attention over a 256-token generation canvas and iteratively denoises that block. When a canvas is finished, the encoder appends it to the cache before the next canvas starts. This is block-autoregressive discrete diffusion, not a single pass over an entire document or ordinary next-token decoding.^{[13]Reference 13DiffusionGemma Model Cardhttps://ai.google.dev/gemma/docs/diffusiongemma/model_card}

The documented sampler allows up to 48 steps per canvas. Its adaptive stopping requires both low average entropy and identical predictions in two consecutive rounds; Google reports typical completion in 12 to 16 steps. Each round retains a low-entropy selection and renoises unselected positions. Selection within a round isn't the same as publishing irrevocable text to a user.^{[1]Reference 1DiffusionGemmahttps://ai.google.dev/gemma/docs/diffusiongemma}

Google also notes that the speed advantage is strongest at low-to-medium batch sizes on a single accelerator. High-QPS cloud batching can shrink that gap.

The important systems difference is dependency shape. Autoregressive decoding has a hard sequential dependency between adjacent generated tokens. Text diffusion still needs multiple model passes, but each pass can update many uncertain positions in one block.

This motivates experiments with block generation, infilling, and editing. It doesn't imply higher saturated server throughput. How long should the canvas be, how many refinement rounds are enough, and how should the UI present text that isn't final?

The state trace below isn't a language model or a DiffusionGemma sampler. A fixed plan fills the last position before the middle and then revises it. It doesn't implement confidence selection, renoising, or learning. The general lesson is the distinction between a provisional canvas state and committed output; a selected implementation determines which positions may change.

import json

canvas = ["<mask>"] * 4
update_plan = [
    {"round": 1, "updates": {0: "write", 3: "later"}},
    {"round": 2, "updates": {1: "unit", 3: "first"}},
    {"round": 3, "updates": {2: "tests"}},
]

trace = []
for step in update_plan:
    for position, token in step["updates"].items():
        canvas[position] = token
    trace.append({
        "round": step["round"],
        "canvas": " ".join(canvas),
        "remaining_masks": canvas.count("<mask>"),
    })

assert trace[0]["canvas"].endswith("later")
assert trace[1]["canvas"].endswith("first")
assert canvas == ["write", "unit", "tests", "first"]
print(json.dumps(trace, indent=2))

[
  {
    "round": 1,
    "canvas": "write <mask> <mask> later",
    "remaining_masks": 2
  },
  {
    "round": 2,
    "canvas": "write unit <mask> first",
    "remaining_masks": 1
  },
  {
    "round": 3,
    "canvas": "write unit tests first",
    "remaining_masks": 0
  }
]

For a UI that promises immutable final text, hold this fixture until its final round. A UI that displays earlier states needs an explicit draft label and replacement behavior. Confidence and stable predictions are sampler stopping signals, not proof of factual correctness; separately check the completed answer or generated code. Measure time to first committed text as well as completion time, since a fast block's total token rate doesn't tell you when its first usable text arrives.

In the masked-token approach here, corruption is discrete and the sampler decides which positions remain uncertain. Text diffusion also includes continuous approaches: Diffusion-LM denoises Gaussian vectors into word vectors.^{[14]Reference 14Diffusion-LM Improves Controllable Text Generationhttps://arxiv.org/abs/2205.14217} Check the representation and length policy of the selected model before borrowing an image sampler. The shared idea is denoising; conditioning and serving constraints differ.

Classifier-free guidance (CFG)

How CFG steers generation without a classifier

How does the model know to draw an "oak dining table" and not a "red running shoe"? Early conditional diffusion systems used classifier guidance. They trained a separate image classifier on noisy inputs, then added the classifier's gradient into the sampling step to push the image toward a target label. That worked, but it required training and maintaining an extra classifier.

CFG (Classifier-Free Guidance) removes that extra classifier. One denoiser learns both conditional and unconditional predictions, then inference extrapolates from the unconditional direction toward the text-conditioned one.^{[15]Reference 15Classifier-Free Diffusion Guidance.https://arxiv.org/abs/2207.12598}

During training, randomly drop the conditioning, such as a text prompt, with probability $p_{uncond}$. Ho and Salimans compared 10%, 20%, and 50% on 64x64 ImageNet and found 10% and 20% worked about equally well.^{[15]Reference 15Classifier-Free Diffusion Guidance.https://arxiv.org/abs/2207.12598}

In real text-to-image systems, the unconditional branch is usually an empty-prompt embedding or learned null token, not a literal all-zero vector. The demo uses a higher drop rate so a four-row batch shows both branches:

import json
import random

def cfg_conditioning_batch(prompts: list[str], drop_prob: float, seed: int = 7) -> list[dict[str, str]]:
    rng = random.Random(seed)
    rows = []
    for prompt in prompts:
        dropped = rng.random() < drop_prob
        rows.append({
            "original_prompt": prompt,
            "conditioning_used": "<null>" if dropped else prompt,
            "branch": "unconditional" if dropped else "conditional",
        })
    return rows

batch = cfg_conditioning_batch(
    [
        "oak dining table in soft light",
        "oak dining table in natural light",
        "mobile dashboard empty state",
        "annotated latency chart hero image",
    ],
    drop_prob=0.35,
)

print(json.dumps(batch, indent=2))

[
  {
    "original_prompt": "oak dining table in soft light",
    "conditioning_used": "<null>",
    "branch": "unconditional"
  },
  {
    "original_prompt": "oak dining table in natural light",
    "conditioning_used": "<null>",
    "branch": "unconditional"
  },
  {
    "original_prompt": "mobile dashboard empty state",
    "conditioning_used": "mobile dashboard empty state",
    "branch": "conditional"
  },
  {
    "original_prompt": "annotated latency chart hero image",
    "conditioning_used": "<null>",
    "branch": "unconditional"
  }
]

During sampling, combine the conditional and unconditional predictions. Common guidance scales above $1$ extrapolate beyond the conditional prediction:

$\hat{\epsilon} = \epsilon_\theta(x_t, t, \emptyset) + w \cdot (\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset))$

Here, $w$ is the guidance scale. When $w=1$, the equation reduces to the conditional prediction $\epsilon_\theta(x_t, t, c)$.

When $w>1$, the model amplifies the difference between conditional and unconditional predictions. It pushes the image away from generic features and toward the text prompt.

Watch the convention. This is the guidance-scale form used in Stable Diffusion and the diffusers library, where $w=1$ means no extra steering. The original paper writes the same thing as $\hat{\epsilon} = (1 + w')\epsilon_\theta(x_t, t, c) - w' \epsilon_\theta(x_t, t, \emptyset)$, where $w'=0$ means no steering.^{[15]Reference 15Classifier-Free Diffusion Guidance.https://arxiv.org/abs/2207.12598} The two map exactly via $w = 1 + w'$, so a reported guidance scale of $7.5$ here corresponds to $w' = 6.5$ in the paper's notation.

These are noise vectors, not verbal descriptions of patches. The conditional-minus-unconditional vector supplies a direction; $w=7.5$ travels 7.5 times that displacement from the unconditional point. Whether the final image better matches oak grain is an empirical outcome, not an interpretation of one noise coordinate.

The practical implementation below shows CFG extrapolation on short vectors. Real systems can batch unconditional and conditional inputs into one forward pass, then split the predictions before applying the same math.

import json

def guided_noise(
    unconditional: list[float],
    conditional: list[float],
    guidance_scale: float,
) -> list[float]:
    return [
        round(base + guidance_scale * (target - base), 3)
        for base, target in zip(unconditional, conditional, strict=True)
    ]

noise_uncond = [0.20, -0.10, 0.05]
noise_cond = [0.05, -0.35, 0.20]

rows = [
    {
        "guidance_scale": scale,
        "guided_noise": guided_noise(noise_uncond, noise_cond, scale),
    }
    for scale in [0.0, 1.0, 7.5, 15.0]
]

print(json.dumps(rows, indent=2))

[
  {
    "guidance_scale": 0.0,
    "guided_noise": [
      0.2,
      -0.1,
      0.05
    ]
  },
  {
    "guidance_scale": 1.0,
    "guided_noise": [
      0.05,
      -0.35,
      0.2
    ]
  },
  {
    "guidance_scale": 7.5,
    "guided_noise": [
      -0.925,
      -1.975,
      1.175
    ]
  },
  {
    "guidance_scale": 15.0,
    "guided_noise": [
      -2.05,
      -3.85,
      2.3
    ]
  }
]

These rows describe the equation with an empty condition as its base. A negative_prompt can replace that base with a negative-conditioned prediction, so it's no longer unconditional. Some pipelines disable the two-branch path when guidance_scale <= 1; passing zero to such an API doesn't request the mathematical $w=0$ result. Check the chosen pipeline rather than inferring API behavior from the formula.^{[16]Reference 16Stable Diffusion: Text-to-imagehttps://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img}

Ordinary CFG represents two per-image predictions per step, even when one batched network call computes both. A network call usually launches many kernels. Distillation can remove the separate branches, while batching mainly changes execution efficiency and memory use.

Faster sampling with implicit diffusion

Denoising Diffusion Implicit Models (DDIM) reuse a diffusion model's training objective while sampling across a selected subset of timesteps. The practical question is whether fewer evaluations preserve the route's required detail and control. A deterministic setting can also avoid fresh noise at each reverse step.

Compare denoiser-evaluation budgets, not just labels. The original DDPM setup is commonly demonstrated with about 1000 reverse steps. A DDIM deployment may try a sparse schedule such as 20 or 50 steps, while distilled or consistency models target still fewer.

DDIM constructs a non-Markovian forward family with the same per-timestep marginals and training objective used by DDPM. Its reverse trajectory still updates one selected state into the next selected state.

DDIM lets the sampler skip steps. Instead of sampling $t, t-1, t-2$, it can step $t, t-\Delta, t-2\Delta$. This can reduce denoiser evaluations substantially.

Here $\hat{x}_0$ is the model's clean-image estimate, $\epsilon_\theta(x_t,t)$ is the predicted noise direction, and $\sigma_t z$ injects randomness. That term vanishes for deterministic DDIM when $\sigma_t = 0$.

For a selected jump from $t$ to $s<t$, the paper parameterizes this noise scale as:

$\sigma_{t\to s}(\eta)=\eta\sqrt{\frac{1-\bar{\alpha}_s}{1-\bar{\alpha}_t}}\sqrt{1-\frac{\bar{\alpha}_t}{\bar{\alpha}_s}}$

Using $0\leq\eta\leq1$ keeps the displayed direction coefficient real for a valid reverse schedule. $\eta=0$ removes newly injected noise; a positive value permits stochastic reverse updates away from the clean endpoint.^{[17]Reference 17Denoising Diffusion Implicit Models.https://arxiv.org/abs/2010.02502} This is a sampler setting, separate from the random initial state and hardware reproducibility.

When the network is trained for $\epsilon$-prediction (as in the DDPM objective above), recover the clean-image estimate before the DDIM update:

That identity bridges the training target $\epsilon_\theta$ to the $\hat{x}_0$ term in the sampler equation. Indices $t$ and $t-1$ in the display are adjacent points on the selected schedule; they need not be unit steps when you skip.

You can train on 1000 noise levels but sample on a sparse subset during deployment. Training and inference schedules don't need to be identical.

Deterministic doesn't mean exactly invertible with a finite-step learned solver. Inversion and editing need their own reconstruction checks. To test the DDIM algebra independently of model quality, reuse the known $x_0=1$ and $\epsilon=-0.4$ from the first example. Recover $x_0$ at $\bar{\alpha}_t=0.5$, then jump directly to $\bar{\alpha}_s=0.9$:

import math

x0, epsilon = 1.0, -0.4
alpha_t, alpha_s = 0.5, 0.9
xt = math.sqrt(alpha_t) * x0 + math.sqrt(1 - alpha_t) * epsilon
x0_hat = (xt - math.sqrt(1 - alpha_t) * epsilon) / math.sqrt(alpha_t)
xs = math.sqrt(alpha_s) * x0_hat + math.sqrt(1 - alpha_s) * epsilon
expected = math.sqrt(alpha_s) * x0 + math.sqrt(1 - alpha_s) * epsilon
assert math.isclose(xs, expected, abs_tol=1e-12)

# Verify the v-parameterization uses a different prediction target.
a, b = math.sqrt(alpha_t), math.sqrt(1 - alpha_t)
v = a * epsilon - b * x0
assert math.isclose(a * xt - b * v, x0, abs_tol=1e-12)
wrong_x0 = (xt - b * v) / a  # scheduler wrongly treats v as epsilon
print(f"xt={xt:.3f}, recovered_x0={x0_hat:.3f}, xs={xs:.3f}")
print(f"v={v:.3f}, wrong_x0_if_target_mismatched={wrong_x0:.3f}")

xt=0.424, recovered_x0=1.000, xs=0.822
v=-0.990, wrong_x0_if_target_mismatched=1.590

The known-noise check should reach $x_s\approx0.822$ and recover $x_0=1$. Feeding $v$ to the epsilon formula instead gives about 1.590. In an actual sample neither the clean endpoint nor true noise is known; the trained prediction supplies that estimate.

Each sampling step can require more than one denoiser prediction. Ordinary CFG still needs both branches when batched; a guidance-distilled model can avoid them. Count work before claiming an interactive route is affordable:

import json

routes = [
    {"name": "offline_ddim_cfg", "steps": 50, "predictions_per_step": 2},
    {"name": "interactive_distilled", "steps": 6, "predictions_per_step": 1},
]

budget = [
    {
        "route": route["name"],
        "denoiser_evaluations": route["steps"] * route["predictions_per_step"],
        "quality_gate_required": True,
    }
    for route in routes
]

print(json.dumps(budget, indent=2))

[
  {
    "route": "offline_ddim_cfg",
    "denoiser_evaluations": 100,
    "quality_gate_required": true
  },
  {
    "route": "interactive_distilled",
    "denoiser_evaluations": 6,
    "quality_gate_required": true
  }
]

Latent diffusion

Architecture

Suppose the denoiser takes the same number of steps in two representations. Pixel-space diffusion processes high-resolution feature maps such as $512 \times 512 \times 3$ at every step. Latent Diffusion Models (LDMs), like Stable Diffusion, move that loop into the latent space of a high-capacity autoencoder.

A variational autoencoder has two parts: an encoder that compresses an image into a compact latent representation and a decoder that reconstructs the image from it. This is learned lossy compression. It preserves enough perceptual structure for the training objective, while reconstruction errors remain part of final output quality.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}

This is the continuous-latent branch of the autoencoder family. A VQ-VAE uses a discrete codebook instead: each patch is snapped to a learned code ID, making the compressed image look more like a sequence of tokens.^{[18]Reference 18Neural Discrete Representation Learning.https://arxiv.org/abs/1711.00937} That design suits a Transformer trained to predict image tokens autoregressively or with masking.

Latent diffusion adds noise to a continuous compressed tensor. This doesn't require every autoencoder to use KL regularization: the original LDM paper also studies VQ regularization with quantization inside the decoder, so diffusion still operates on the continuous encoder output.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752} Diffusing this tensor is different from corrupting discrete code IDs; check which representation the denoiser actually receives.

During training, the VAE encoder maps an image to a clean latent $z_0$, and the model learns to denoise noisy versions of that latent.

For pure text-to-image inference, there's no input image. Start from random latent noise $z_T \sim \mathcal{N}(0, \mathbf{I})$, run the reverse process in latent space, and decode at the end.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752} Image-to-image and inpainting also encode source images and use their own noise/mask rules.

The minimal inference path decodes only the final latent. Intermediate previews may decode a scheduler's clean-latent estimate at extra cost. If the denoiser predicts noise or velocity, the scheduler uses it to update $z_t \rightarrow z_{t-1}$; predicted noise isn't itself a clean latent. Apply the checkpoint's latent scaling convention before the VAE decoder.

Why latent space?

Moving diffusion from pixels to latents splits the workload into two phases. The autoencoder compresses and reconstructs pixels with measurable detail loss. The diffusion model performs repeated denoising on the smaller latent state.

A $64 \times 64 \times 4$ latent tensor has $48\times$ fewer scalar values than a $512 \times 512 \times 3$ image. That is the Stable Diffusion 1.x-style shape used in the worked count below, not a universal latent.

Later systems can change channel count or spatial compression; Stable Diffusion 3 reports a 16-channel autoencoder.^{[9]Reference 9Scaling Rectified Flow Transformers for High-Resolution Image Synthesishttps://arxiv.org/abs/2403.03206} Count the latent values the loop actually iterates on, then benchmark the resulting route.

The same latent has $64\times$ fewer spatial locations. Neither ratio is a promised FLOP, memory, or latency speedup: hidden channels, attention, network depth, and decoding all contribute work. Evaluate VAE reconstruction separately to distinguish compression loss from denoiser errors.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}

import json
import math

pixel_shape = (512, 512, 3)
latent_shape = (64, 64, 4)
pixel_values = math.prod(pixel_shape)
latent_values = math.prod(latent_shape)

print(json.dumps({
    "pixel_values": pixel_values,
    "latent_values": latent_values,
    "scalar_reduction": pixel_values // latent_values,
    "spatial_reduction": (pixel_shape[0] * pixel_shape[1]) // (latent_shape[0] * latent_shape[1]),
    "throughput_requires_benchmark": True,
}, indent=2))

{
  "pixel_values": 786432,
  "latent_values": 16384,
  "scalar_reduction": 48,
  "spatial_reduction": 64,
  "throughput_requires_benchmark": true
}

Text conditioning via cross-attention

Text-conditioning recipes are model-specific. The latent diffusion paper introduces cross-attention conditioning; Stable Diffusion uses a pretrained CLIP text encoder.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752[16]Reference 16Stable Diffusion: Text-to-imagehttps://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img} Other generators may use T5-family encoders, multiple text encoders, or different freeze/train choices.

Prompt embeddings condition repeated denoising updates; freeze choices depend on the selected training recipe.

Cross-attention projects flattened image or latent features into queries, and text embeddings into keys and values. A numerically stable softmax turns compatibility scores into weights. The calculation below starts with already-projected vectors: one three-dimensional query attends to three keys and mixes their two-dimensional values. Learned projections, heads, residuals, and normalization are omitted.

import json
import math

def dot(left: list[float], right: list[float]) -> float:
    return sum(a * b for a, b in zip(left, right, strict=True))

def softmax(values: list[float]) -> list[float]:
    largest = max(values)
    exp_values = [math.exp(value - largest) for value in values]
    total = sum(exp_values)
    return [value / total for value in exp_values]

latent_query = [0.25, 0.10, 0.70]
text_tokens = [
    {"token": "oak", "key": [0.30, 0.05, 0.65], "value": [0.70, 0.20]},
    {"token": "dining", "key": [0.10, 0.80, 0.10], "value": [0.20, 0.70]},
    {"token": "table", "key": [0.35, 0.10, 0.55], "value": [0.80, 0.10]},
]

logits = [
    dot(latent_query, token["key"]) / math.sqrt(len(latent_query))
    for token in text_tokens
]
weights = softmax(logits)
conditioned_patch = [
    sum(weight * token["value"][i] for weight, token in zip(weights, text_tokens, strict=True))
    for i in range(2)
]

print(json.dumps({
    "attention_weights": {
        token["token"]: round(weight, 3)
        for token, weight in zip(text_tokens, weights, strict=True)
    },
    "conditioned_patch": [round(value, 3) for value in conditioned_patch],
}, indent=2))

{
  "attention_weights": {
    "oak": 0.359,
    "dining": 0.292,
    "table": 0.349
  },
  "conditioned_patch": [
    0.589,
    0.311
  ]
}

Control and specialization

Once the base route works, teams usually add targeted controls around it instead of retraining the whole backbone.

• LoRA (Low-Rank Adaptation): Full fine-tuning is expensive. LoRA^{[19]Reference 19LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} inserts small trainable matrices into selected layers so you can specialize a base model with a much smaller parameter update.
• Inpainting and outpainting: The model conditions on known pixels and a mask, which lets it fill holes, replace objects, or extend the canvas while preserving local context.
• ControlNet / adapters: ControlNet^{[20]Reference 20Adding Conditional Control to Text-to-Image Diffusion Models.https://arxiv.org/abs/2302.05543} freezes the pretrained backbone and adds trainable control branches for signals like depth, edges, or pose. It steers toward supplied structure; it doesn't guarantee exact geometric compliance. Measure how closely the generated pose or edges match the control.

Fast inference in practice

DDIM shows the first big shortcut. A production serving pipeline can combine several optimizations, but only after profiling each route. Treat every step count below as a reported or representative route budget, not an SLA. Record checkpoint, resolution, batch or concurrency, hardware, precision, guidance, warmup, end-to-end latency, and quality gates before comparing candidates.

1. Sampler choice (DDIM / DPM-Solver): The original DDPM reference uses a long stochastic chain. DDIM permits a sparse schedule, such as a 20- or 50-step candidate, without retraining the denoiser.^{[17]Reference 17Denoising Diffusion Implicit Models.https://arxiv.org/abs/2010.02502} DPM-Solver exploits the structure of the diffusion ordinary differential equation (ODE) to integrate with fewer function evaluations.^{[21]Reference 21DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepshttps://arxiv.org/abs/2206.00927} Match the solver to the model's prediction target and noise schedule.
2. Latent Consistency Models (LCM): Consistency models learn to map points on one probability-flow ODE trajectory to the same nearly clean endpoint.^{[22]Reference 22Consistency Models.https://arxiv.org/abs/2303.01469} LCM applies related distillation in latent space and reports 2 to 4-step synthesis.^{[23]Reference 23Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.https://arxiv.org/abs/2310.04378} LCM-LoRA packages acceleration as low-rank adapters that can combine with compatible fine-tuned checkpoints.^{[24]Reference 24LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulehttps://arxiv.org/abs/2311.05556} It still requires matching model families and scheduler settings.
3. Adversarial Diffusion Distillation (ADD / SDXL Turbo): Progressive distillation halves required steps by training student models to match two teacher steps at once.^{[6]Reference 6Progressive Distillation for Fast Sampling of Diffusion Models.https://arxiv.org/abs/2202.00512} Adversarial Diffusion Distillation (ADD) combines score distillation from a diffusion teacher with an adversarial discriminator loss. Its paper reports single-step to 4-step image synthesis, including SDXL-scale comparisons.^{[25]Reference 25Adversarial Diffusion Distillation.https://arxiv.org/abs/2311.17042}
4. Flow matching: This changes the training target and integration path.^{[11]Reference 11Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowhttps://arxiv.org/abs/2209.03003[10]Reference 10Flow Matching for Generative Modelinghttps://arxiv.org/abs/2210.02747} It isn't a scheduler swap that makes any DDPM model a few-step flow model. Test the checkpoint's supported sampling recipe.
5. Guidance and CFG distillation: Distilling classifier-free guidance into a student denoiser can remove separate conditional and unconditional predictions at runtime. If one student prediction replaces two equivalent-quality predictions, denoiser compute changes from $2\times$ to $1\times$ per step. Verify that equivalence on the release fixture.
6. Kernel and hardware efficiency: Transformer and attention-heavy denoisers can benefit from fused kernels such as FlashAttention,^{[26]Reference 26FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135} FP8/INT8 weight quantization, and graph-level optimizations such as TensorRT or torch.compile. Measure memory, warmup, throughput, p50, and p95 because a kernel speedup need not improve end-to-end latency.

Offline concept boards may tolerate slower samplers; interactive previews need measurements against their stricter latency budget. No step count makes output faithful to a real screenshot.

Common pitfalls

"CFG scale is a probability"

• Symptom: A team sets $w=7.5$ because it "sounds confident," then gets oversaturated textures and repeated details.
• Cause: $w$ is an extrapolation coefficient between unconditional and conditional noise, not a chance the prompt is followed.
• Fix: Measure adherence, diversity, and artifacts on the target prompt slice. $w=0$ is unconditional, $w=1$ is plain conditional, and larger values overshoot.^{[15]Reference 15Classifier-Free Diffusion Guidance.https://arxiv.org/abs/2207.12598}

"The VAE decoder consumes predicted noise"

• Symptom: Debugging dumps look like static, or operators think every denoiser step should produce a viewable image.
• Cause: If the model predicts $\epsilon$ or $v$, the scheduler converts that into the next latent or a clean-latent estimate.^{[4]Reference 4Denoising Diffusion Probabilistic Models.https://arxiv.org/abs/2006.11239[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}
• Fix: Check prediction type and VAE scaling. Decode the final latent, or an explicit clean estimate for a preview, not raw noise predictions.

"Latent diffusion removes the autoencoder bottleneck"

• Symptom: Fine text, logos, and UI chrome stay mushy even after more denoising steps.
• Cause: Lossy compression can remove detail, but denoising, conditioning, and training data can also cause these errors.
• Fix: Reconstruct held-out real images through the VAE first. If the same detail fails before diffusion, more sampling steps won't repair that bottleneck.^{[7]Reference 7High-Resolution Image Synthesis with Latent Diffusion Models.https://arxiv.org/abs/2112.10752}

"DiT is automatically cheaper"

• Symptom: A transformer backbone is chosen for latency, then token count blows up the attention bill.
• Cause: DiT scaling results are about quality versus compute, not a promise of cheaper serving.
• Fix: Patch in latent space, budget tokens, and measure the actual route. Joint text-image attention, as in SD3's MM-DiT, is a published recipe, not a universal speedup.^{[8]Reference 8Scalable Diffusion Models with Transformers.https://arxiv.org/abs/2212.09748[9]Reference 9Scaling Rectified Flow Transformers for High-Resolution Image Synthesishttps://arxiv.org/abs/2403.03206}

"Text diffusion is next-token decoding with a different sampler"

• Symptom: Serving assumes tokens stream left to right, or treats a missing canvas plan as a tokenizer detail.
• Cause: Discrete diffusion refines a block. DiffusionGemma is encoder-decoder and block-autoregressive: a 256-token canvas is denoised with bidirectional attention, then appended before the next canvas starts.^{[1]Reference 1DiffusionGemmahttps://ai.google.dev/gemma/docs/diffusiongemma[13]Reference 13DiffusionGemma Model Cardhttps://ai.google.dev/gemma/docs/diffusiongemma/model_card}
• Fix: Plan canvas length, refinement rounds, commit confidence, and UI behavior for text that can change before it's final.

"A missing watermark proves the image is human-made"

• Symptom: A copied file without synthetic=true is treated as photographic evidence.
• Cause: API labels don't travel with the bytes, and detectors can be uncertain after edits.
• Fix: Embed a watermark, bind the asset digest to the job record, and verify the distributed derivative before publishing. Nondetection still isn't proof of human origin.

Teams building design-asset pipelines can't treat synthetic imagery as verified UI or incident evidence. Generated outputs can show incorrect labels, brand marks, chart values, legal text, or unsafe content.

High-risk routes need output filtering, provenance labeling, watermark verification, and human review before publication. Test the release decision with supplied audit results below. One variant passes; one needs review; one carries evidence for different bytes; another comes from a superseded worker. This function doesn't implement a detector, classifier, object store, or atomic database update.

import json

def release_decision(record: dict, current_attempt: int) -> str:
    if type(current_attempt) is not int or current_attempt < 1:
        raise ValueError("current_attempt must be a positive integer")
    attempt = record.get("attempt")
    if type(attempt) is not int or attempt != current_attempt:
        return "stale_attempt"
    if record.get("policy") != "pass" or record.get("human_review") is not False:
        return "review"
    if record.get("watermark") != "detected" or record.get("manifest_committed") is not True:
        return "hold"
    digest = record.get("asset_digest")
    if not isinstance(digest, str) or not digest or record.get("checked_digest") != digest:
        return "hold"
    return "publish_checked_bytes"

base = {
    "attempt": 2, "policy": "pass", "human_review": False,
    "watermark": "detected", "manifest_committed": True,
    "asset_digest": "digest-A", "checked_digest": "digest-A",
}
fixtures = [
    ("background", base),
    ("logo_requires_review", {**base, "human_review": True}),
    ("unchecked_resize", {**base, "asset_digest": "digest-B"}),
    ("expired_worker", {**base, "attempt": 1}),
    ("missing_detector_result", {**base, "watermark": None}),
    ("string_manifest_verdict", {**base, "manifest_committed": "false"}),
    ("float_attempt", {**base, "attempt": 2.0}),
    ("nonstring_digests", {**base, "asset_digest": [1], "checked_digest": [1]}),
]
decisions = [
    {"asset": name, "decision": release_decision(record, current_attempt=2)}
    for name, record in fixtures
]

assert release_decision(base, 2) == "publish_checked_bytes"
assert release_decision({**base, "manifest_committed": "false"}, 2) == "hold"
assert release_decision({**base, "manifest_committed": False}, 2) == "hold"
assert release_decision({**base, "attempt": 2.0}, 2) == "stale_attempt"
assert release_decision({**base, "asset_digest": [1], "checked_digest": [1]}, 2) == "hold"
print(json.dumps(decisions, indent=2))

[
  {
    "asset": "background",
    "decision": "publish_checked_bytes"
  },
  {
    "asset": "logo_requires_review",
    "decision": "review"
  },
  {
    "asset": "unchecked_resize",
    "decision": "hold"
  },
  {
    "asset": "expired_worker",
    "decision": "stale_attempt"
  },
  {
    "asset": "missing_detector_result",
    "decision": "hold"
  },
  {
    "asset": "string_manifest_verdict",
    "decision": "hold"
  },
  {
    "asset": "float_attempt",
    "decision": "stale_attempt"
  },
  {
    "asset": "nonstring_digests",
    "decision": "hold"
  }
]

The original truthiness check would accept the nonempty string "false" as a committed manifest, and Python equality would accept attempt 2.0 as integer 2. The repaired fixture requires an exact boolean verdict and integer attempt, plus a matching nonempty string digest. These checks validate record representation. A trusted current-attempt lookup, authorized detector/policy records, actual digest calculation, and atomic publication still belong to the real backend.

Run one generation release audit

A release audit asks whether a candidate improves accepted assets, not whether one sample looks good. Pin model, VAE, scheduler, seed, prompt, negative prompt, step count, and guidance scale.

Vary one axis at a time. Record prompt alignment, text and logo corruption, memorization risk, safety-filter outcomes, latency, and provenance metadata in one release scorecard over the same fixture set.

Use the failure signal to choose the next check. Prompt alignment with garbled text points to representation or VAE limits. Good samples with a p95 breach point to evaluation count, batching, or release work. Uncertain watermark detection blocks publication until the final derivative is checked.

High-impact screenshots, legal text, brand assets, or incident evidence stay behind human review even when automated checks pass.

Review may approve a clearly labeled illustrative depiction. It can't turn a generated latency chart into an observed incident measurement; use authenticated source artifacts for that factual claim.

Final design checklist

Use this rubric to review your service diagram and a recorded generation trace. A calculation alone doesn't establish a deployable service.

• Mechanism: Compute one forward sample and one DDIM update; explain what fails when a velocity prediction is treated as epsilon.
• Capacity: Account for variants, steps, CFG branches, batching compatibility, queue age, and release work without treating prediction counts as GPU benchmarks.
• Recovery: Bind tenant and request hash at admission; fence superseded workers and preserve per-variant random state and scheduler state where resumption is supported.
• Publication: Authorize asset access, inspect final distributed bytes, and commit the matching digest and manifest before returning a URL.
• Evaluation: Compare accepted quality, safety, diversity, controls, latency, and cost on fixed prompt slices; separate VAE reconstruction errors from denoiser errors.

Follow-up questions

The CPU exercises validate formulas and release decisions. A real model release still needs GPU measurements, image-quality review, detector calibration, tenant-isolation tests, and crash/retry tests against the actual storage and queue backends.