LearnPreparation & PrerequisitesThe Transformer Architecture End-to-End

🧠EasyTransformer Architecture

The Transformer Architecture End-to-End

Trace an incident note through masked attention, a decoder block, and next-token logits with readable NumPy and PyTorch code.

13 min read

Learning path

Step 24 of 177 in the full curriculum

Autoencoders and VAEs Language Modeling & Next Tokens

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

An autoencoder turned an input patch into a compact representation. A Transformer must build a useful representation at every token position, because an incident note such as Key blocked by depends on what came earlier and must still predict what comes next.

Follow a small decoder-only Transformer forward pass. The dimensions stay tiny enough to print, then the same operations get assembled in PyTorch. Earlier lessons introduced cross-entropy; the next chapter applies it to shifted next-token targets across a sequence.

Decoder-only Transformer step showing a three-token feature matrix, a causal attention heatmap inside repeated decoder blocks, a same-shaped contextual matrix, vocabulary logits with policy highest, and policy appended to form a four-token context. — Masked blocks preserve one feature row per token while refining each row with visible context. The final row scores the vocabulary; appending the selected token grows the matrix for the next step.

The Transformer introduced in 2017 used an encoder and a decoder for translation. Its decoder combines masked self-attention, encoder-decoder attention, and a feed-forward network. A text-generation decoder-only model keeps the causal self-attention path and predicts the next token from preceding context.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}^{[2]Reference 2Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}

Tokens become positioned vectors

Use a simplified incident-policy vocabulary. We won't claim these are IDs from a production tokenizer; we're choosing IDs deliberately so the forward pass is readable:

Text so far	Token IDs	Desired continuation
`Key blocked by`	`[0, 1, 2]`	likely `policy`

A model doesn't operate on token labels. It looks up one learned vector per ID, then adds or applies position information so blocked by differs from by blocked. In this first calculation, position is a small vector added to each token embedding.

positioned-token-vectors.py

import numpy as np

tokens = ["Key", "blocked", "by"]
ids = np.array([0, 1, 2])
embedding_table = np.array([
    [1.0, 0.0, 0.4, 0.0],  # Key
    [0.2, 1.0, 0.5, 0.0],  # blocked
    [0.0, 0.3, 0.8, 1.0],  # by
    [0.1, 0.5, 1.0, 0.9],  # policy
])
position = np.array([
    [0.00, 0.00, 0.00, 0.00],
    [0.05, 0.00, 0.00, 0.05],
    [0.10, 0.00, 0.00, 0.10],
])

x = embedding_table[ids] + position
print("tokens:", tokens)
print("ids shape:", ids.shape)
print("hidden shape:", x.shape)
print("last positioned vector:", np.round(x[-1], 2))

Output

tokens: ['Key', 'blocked', 'by']
ids shape: (3,)
hidden shape: (3, 4)
last positioned vector: [0.1 0.3 0.8 1.1]

Our tiny model uses three tokens and four features, so its hidden-state matrix has shape [3, 4]. Real models use far wider vectors and batches, but the axes mean the same thing:

Axis	Tiny example	General name
Number of sequences	omitted	`B` (batch size)
Token positions	`3`	`T` (sequence length)
Features per token	`4`	`d_model` (model width)

Diagram showing Positioned vectors Key blocked by [T, d_model], N masked blocks [T, d_model], Final norm + head Last-position logits [vocab], and Next token policy. — Positioned vectors Key blocked by [T, d_model], N masked blocks [T, d_model], Final norm + head Last-position logits [vocab], and Next token policy.

Attention lets a token read earlier tokens

An RNN transports information through one recurrent state. Self-attention takes a different route: at each position it computes relevance scores against other visible positions, then blends their value vectors. For a decoder, "visible" means the current and earlier tokens only.

Each token produces three learned projections:

A query asks what information this position needs.
A key advertises what a position contains.
A value supplies information if the query matches its key.

The projections come from the same hidden-state matrix x, using different learned weights W_q, W_k, and W_v.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

To see the multiplication, take one two-feature attention head. These arrays stand in for outputs of its learned projections:

query-key-scores.py

import numpy as np

q = np.array([
    [1.0, 0.0],  # Key asks mainly for credential context
    [0.5, 1.0],  # blocked asks for credential and action context
    [0.2, 1.2],  # by asks strongly for policy context
])
k = np.array([
    [1.0, 0.0],
    [0.4, 1.0],
    [0.0, 0.8],
])
v = np.array([
    [1.0, 0.0],
    [0.3, 1.0],
    [0.0, 0.6],
])

raw_scores = q @ k.T
print("Q shape:", q.shape, "K shape:", k.shape, "V shape:", v.shape)
print("Q @ K.T shape:", raw_scores.shape)
print("raw score row for 'by':", np.round(raw_scores[2], 2))

Output

Q shape: (3, 2) K shape: (3, 2) V shape: (3, 2)
Q @ K.T shape: (3, 3)
raw score row for 'by': [0.2  1.28 0.96]

Three queries compared with three keys produce a [3, 3] matrix. The square T x T attention-score matrix is why full attention becomes expensive as context length grows: doubling T gives four times as many pair scores.

Scale before softmax

Scaled dot-product attention divides query-key scores by the square root of the head width, then applies softmax:

\operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Here, d_k is the number of features in one key or query vector. Vaswani et al. explain that without scaling, larger dot products push softmax toward very small gradients when the key width is large.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

This calculation uses a 64-feature head and compares unscaled and scaled scores for one token:

why-scale-attention.py

import numpy as np

def softmax(values):
    shifted = values - values.max()
    exp = np.exp(shifted)
    return exp / exp.sum()

scores = np.array([12.4, 9.1, 3.8])
scaled = scores / np.sqrt(64)

print("unscaled weights:", np.round(softmax(scores), 3))
print("scaled scores:", np.round(scaled, 3))
print("scaled weights:", np.round(softmax(scaled), 3))

Output

unscaled weights: [0.964 0.036 0.   ]
scaled scores: [1.55  1.138 0.475]
scaled weights: [0.499 0.33  0.17 ]

Without scaling, nearly all attention mass lands on one key. Scaling doesn't prevent a confident choice after training; it prevents the head width alone from making early scores overly sharp.

The causal mask prevents answer leakage

During training, the model can process Key blocked by policy in one matrix operation. Position by is supposed to predict policy, so it must not read the policy representation first. A causal mask sets future scores to negative infinity before softmax; those entries receive probability zero.

Query position	Can read
`Key`	`Key`
`blocked`	`Key`, `blocked`
`by`	`Key`, `blocked`, `by`
`policy`	all four positions

causal-mask.py

import numpy as np

scores = np.array([
    [2.0, 1.5, 0.5, 1.0],
    [1.0, 2.5, 1.5, 0.5],
    [0.5, 1.0, 3.0, 2.0],
    [1.5, 0.5, 1.0, 2.5],
])
future = np.triu(np.ones_like(scores, dtype=bool), k=1)
masked_scores = np.where(future, -np.inf, scores)

shifted = masked_scores - masked_scores.max(axis=-1, keepdims=True)
weights = np.exp(shifted)
weights /= weights.sum(axis=-1, keepdims=True)

print("masked weights:")
print(np.round(weights, 3))
print("future attention mass:", float(weights[future].sum()))

Output

masked weights:
[[1.    0.    0.    0.   ]
 [0.182 0.818 0.    0.   ]
 [0.067 0.111 0.821 0.   ]
 [0.213 0.078 0.129 0.579]]
future attention mass: 0.0

The important output is future attention mass: 0.0. If a causal language model reports nonzero future attention during training, it has a data-leak bug, not an impressive loss curve.

Heads run in parallel; blocks run in sequence

One attention head can learn one set of relevance patterns. Multi-head attention projects the model features into several heads, applies attention in parallel, concatenates their outputs, and projects back to d_model.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

Suppose our tiny model width is four and it has two heads. Each head owns two features. Splitting and merging must preserve every token position:

split-and-merge-heads.py

import numpy as np

batch, tokens, d_model, heads = 1, 3, 4, 2
d_head = d_model // heads
projected = np.arange(batch * tokens * d_model).reshape(batch, tokens, d_model)

split = projected.reshape(batch, tokens, heads, d_head).transpose(0, 2, 1, 3)
merged = split.transpose(0, 2, 1, 3).reshape(batch, tokens, d_model)

print("per-head shape:", split.shape)
print("merged shape:", merged.shape)
print("merge restores values:", bool(np.array_equal(projected, merged)))

Output

per-head shape: (1, 2, 3, 2)
merged shape: (1, 3, 4)
merge restores values: True

The head axis is parallel work inside one block. The block axis is sequential depth: output from block 1 enters block 2. Mixing these concepts leads to configuration and shape mistakes.

A block alternates communication and local transformation

Attention lets token positions exchange information. A feed-forward network (FFN) then transforms each position independently, using the same weights at every position. The original Transformer used a two-layer FFN with a ReLU nonlinearity between the layers.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762}

\operatorname{FFN}(x) = \operatorname{ReLU}(xW_1 + b_1)W_2 + b_2

The first projection commonly widens the feature dimension; the second returns it to d_model, which keeps blocks stackable.

position-wise-ffn.py

import numpy as np

x = np.array([
    [1.0, 0.0, 0.4, 0.0],
    [0.2, 1.0, 0.5, 0.0],
    [0.0, 0.3, 0.8, 1.0],
])
w1 = np.array([
    [1.0, 0.0, 0.5, 0.0, 0.0, 0.2],
    [0.0, 1.0, 0.0, 0.5, 0.2, 0.0],
    [0.4, 0.2, 1.0, 0.0, 0.0, 0.5],
    [0.0, 0.2, 0.0, 1.0, 0.4, 0.0],
])
w2 = np.array([
    [0.3, 0.0, 0.0, 0.1],
    [0.0, 0.3, 0.1, 0.0],
    [0.2, 0.0, 0.3, 0.0],
    [0.0, 0.2, 0.0, 0.3],
    [0.1, 0.0, 0.0, 0.2],
    [0.0, 0.1, 0.2, 0.0],
])

expanded = np.maximum(0.0, x @ w1)
ffn_output = expanded @ w2
print("input shape:", x.shape)
print("expanded shape:", expanded.shape)
print("returned shape:", ffn_output.shape)
print("last-token update:", np.round(ffn_output[-1], 3))

Output

input shape: (3, 4)
expanded shape: (3, 6)
returned shape: (3, 4)
last-token update: [0.302 0.468 0.386 0.469]

Residual paths and normalization

A residual connection adds a sublayer result back to its input. It creates an identity path through deep stacks, following the residual-network idea introduced by He et al.^{[3]Reference 3Deep Residual Learning for Image Recognition.https://arxiv.org/abs/1512.03385} Layer normalization computes statistics within each token vector; its learned scale and offset can then adjust the normalized output.^{[4]Reference 4Layer Normalization.https://arxiv.org/abs/1607.06450}

There are multiple valid placements. The 2017 Transformer applied normalization after each residual addition. The diagram and PyTorch blocks later in this chapter use pre-normalization: normalize first, compute the sublayer update, then add it to the residual stream. The two layouts share attention, FFN, and residual concepts; don't accidentally describe one while coding the other.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762} Xiong et al. connect Pre-LN with better-behaved gradients at initialization and show that it can train without learning-rate warmup in their evaluated tasks.^{[5]Reference 5On Layer Normalization in the Transformer Architecture.https://arxiv.org/abs/2002.04745}

This NumPy calculation demonstrates the pre-normalization order by normalizing a tensor before adding a small simulated update:

pre-normalized-residual.py

import numpy as np

def normalize(x, eps=1e-5):
    mean = x.mean(axis=-1, keepdims=True)
    variance = ((x - mean) ** 2).mean(axis=-1, keepdims=True)
    return (x - mean) / np.sqrt(variance + eps)

x = np.array([
    [1.0, 2.0, 0.0, 1.0],
    [0.2, 0.4, 0.8, 0.6],
])
normalized = normalize(x)
sublayer_update = 0.1 * normalized
residual_output = x + sublayer_update

print("normalized means:", np.round(normalized.mean(axis=-1), 6))
print("normalized variances:", np.round(normalized.var(axis=-1), 4))
print("residual output shape:", residual_output.shape)

Output

normalized means: [0. 0.]
normalized variances: [1.     0.9998]
residual output shape: (2, 4)

Pre-normalized Transformer block drawn as a same-shaped residual stream with two explicit bypass paths: one branch applies LayerNorm and masked attention before an add, and the next applies LayerNorm and a per-token feed-forward network before a second add. — Each pre-LN branch reads a normalized copy of the residual stream, computes an update, and merges it back with an addition. The untouched bypass keeps the tensor shape and identity path available through both sublayers.

Implement masked attention in PyTorch

Now implement a single causal multi-head self-attention layer. The projections are learned linear maps; the mask is created from token positions. masked_fill is the key line: it turns every future score into negative infinity before softmax.

causal-self-attention.py

import math
import torch
from torch import nn

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model=8, heads=2):
        super().__init__()
        assert d_model % heads == 0
        self.heads = heads
        self.d_head = d_model // heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.out = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x):
        batch, tokens, d_model = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)
        def split_heads(tensor):
            return tensor.view(batch, tokens, self.heads, self.d_head).transpose(1, 2)
        q, k, v = map(split_heads, (q, k, v))

        scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_head)
        future = torch.triu(
            torch.ones(tokens, tokens, dtype=torch.bool, device=x.device),
            diagonal=1,
        )
        weights = torch.softmax(scores.masked_fill(future, float("-inf")), dim=-1)
        mixed = weights @ v
        merged = mixed.transpose(1, 2).contiguous().view(batch, tokens, d_model)
        return self.out(merged), weights

torch.manual_seed(4)
x = torch.randn(1, 4, 8)
attention = CausalSelfAttention()
output, weights = attention(x)
future = torch.triu(torch.ones(4, 4, dtype=torch.bool), diagonal=1)

print("attention output shape:", tuple(output.shape))
print("weights shape:", tuple(weights.shape))
print("largest future weight:", f"{weights[0, :, future].max().item():.6f}")

Output

attention output shape: (1, 4, 8)
weights shape: (1, 2, 4, 4)
largest future weight: 0.000000

The output shape matches the input shape, while the [batch, heads, tokens, tokens] weight tensor exposes each head's routing decisions.

Assemble one decoder block

With attention implemented, a small pre-normalized block is short: normalize, attend, add; normalize, run FFN, add. PyTorch's LayerNorm includes the learned scale and offset that the earlier NumPy calculation omitted. This block uses GELU instead of the original Transformer's ReLU, so you can see that the activation choice can change while the FFN's structural role stays the same: widen, transform, and return to d_model.

decoder-block.py

import math
import torch
from torch import nn

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model=8, heads=2):
        super().__init__()
        assert d_model % heads == 0
        self.heads = heads
        self.d_head = d_model // heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.out = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x):
        batch, tokens, d_model = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)
        q = q.view(batch, tokens, self.heads, self.d_head).transpose(1, 2)
        k = k.view(batch, tokens, self.heads, self.d_head).transpose(1, 2)
        v = v.view(batch, tokens, self.heads, self.d_head).transpose(1, 2)
        scores = q @ k.transpose(-2, -1) / math.sqrt(self.d_head)
        mask = torch.triu(
            torch.ones(tokens, tokens, dtype=torch.bool, device=x.device),
            diagonal=1,
        )
        weights = torch.softmax(scores.masked_fill(mask, float("-inf")), dim=-1)
        joined = (weights @ v).transpose(1, 2).contiguous().view(batch, tokens, d_model)
        return self.out(joined)

class DecoderBlock(nn.Module):
    def __init__(self, d_model=8, heads=2, d_ff=24):
        super().__init__()
        self.norm1 = nn.LayerNorm(d_model)
        self.attention = CausalSelfAttention(d_model, heads)
        self.norm2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model),
        )

    def forward(self, x):
        x = x + self.attention(self.norm1(x))
        x = x + self.ffn(self.norm2(x))
        return x

torch.manual_seed(5)
x = torch.randn(2, 4, 8, requires_grad=True)
block = DecoderBlock()
output = block(x)
output.square().mean().backward()

print("input -> output:", tuple(x.shape), "->", tuple(output.shape))
print("input gradient is finite:", bool(torch.isfinite(x.grad).all()))

Output

input -> output: (2, 4, 8) -> (2, 4, 8)
input gradient is finite: True

This block doesn't generate a word yet. It produces contextual vectors with the same [B, T, d_model] shape as its input, so another block can consume them.

Project the final position into a vocabulary

After the final block, many pre-normalized decoder stacks apply one final normalization before the output head. For example, GPT-2 applies its final ln_f immediately before computing logits.^{[6]Reference 6GPT-2 Source Implementation.https://github.com/openai/gpt-2/blob/master/src/model.py} A learned output matrix then converts each hidden vector into one raw score per vocabulary item. These scores are logits. During generation, use only the final position's scores because that position represents all text currently visible to the decoder. The small calculation below starts from an already normalized final-position vector.

next-token-head.py

import numpy as np

vocab = np.array(["policy", "today", "late", "auth", "token", "service"])
last_hidden_normalized = np.array([0.7, -0.2, 0.5, 0.1])
output_weight = np.array([
    [1.714, 0.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, -1.75, 0.0, 0.0, 0.0, 2.0],
    [0.0, 0.0, 0.8, 0.0, 0.24, 0.0],
    [0.0, 0.0, 0.0, 0.5, 0.0, -3.0],
])

logits = last_hidden_normalized @ output_weight
probabilities = np.exp(logits - logits.max())
probabilities /= probabilities.sum()
best = probabilities.argmax()

print("logits shape:", logits.shape)
print("top token:", vocab[best])
print("top probability:", f"{probabilities[best]:.3f}")

Output

logits shape: (6,)
top token: policy
top probability: 0.373

Transformer shape trace showing a final-normalized three-by-four hidden-state matrix, selection of its final four-feature row, multiplication by a four-by-six output-weight matrix, and six vocabulary logits with policy highest. — The decoder stack and final normalization preserve the `[3, 4]` hidden-state shape. Selecting the final row gives four features, and multiplying by a `[4, 6]` output matrix produces one logit for each of six vocabulary candidates.

The result policy completes Key blocked by policy. Conceptually, generate a longer answer by appending that token, running the stack with the longer context, and choosing the next one. Optimized serving reuses earlier attention keys and values through a KV cache instead of recomputing them for every new token. The next chapter connects this forward pass to shifted targets and the cross-entropy objective introduced earlier.

Encoder, decoder, and encoder-decoder visibility

Decoder-only generation is one member of the Transformer family. Architecture choice is largely a question of which positions may read which information:

Variant	Visibility rule	Suitable lesson example	Representative source
Encoder-only	Each input position can read both left and right context	Classify a code review after reading the full note	BERT^{[7]Reference 7BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}
Decoder-only	Each position can read only itself and prior tokens	Generate an incident explanation left to right	GPT^{[2]Reference 2Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}
Encoder-decoder	Encoder reads source; decoder reads source plus prior output tokens	Translate a deployment rollback policy	T5^{[8]Reference 8Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}

Three visibility matrices compare encoder-only full bidirectional attention, decoder-only lower-triangular causal attention, and encoder-decoder attention with a full source block, causal target block, and target-to-source cross-attention. — Encoder-only attention fills the input-to-input matrix. Decoder-only attention keeps only the causal lower triangle. Encoder-decoder attention combines a full source block, a causal target block, and target queries that can read every source position.

In self-attention, queries, keys, and values come from one sequence. In encoder-decoder cross-attention, decoder queries read keys and values produced by the encoder. A plain decoder-only text generator has no separate encoder sequence, so its core path is causal self-attention.

Diagnose a broken forward pass

The fastest way to debug a Transformer is to ask what invariant was violated:

Symptom	Likely cause	Check
Training loss looks suspiciously tiny, but generation fails	Future tokens leaked through attention	Assert masked future attention mass is zero
Head split crashes or silently produces the wrong shape	`d_model` isn't divisible by `n_heads`, or axes were transposed incorrectly	Check `[B, H, T, d_head]` then merge back to `[B, T, d_model]`
Attention works on CPU but fails after moving the block to a GPU	Causal mask stayed on CPU while scores moved to the accelerator	Create the mask on `x.device`
A block can't be stacked with another block	FFN or output projection changed model width too early	Keep block output at `d_model`; project to vocabulary only at the end
Explanation says decoder-only model reads an encoder	Self-attention and cross-attention were confused	Identify whether a separate source sequence exists

Use a deliberately bad mask to make leakage visible:

detect-mask-leakage.py

import numpy as np

scores = np.array([
    [2.0, 5.0, 4.0],
    [1.0, 2.0, 6.0],
    [0.5, 1.0, 2.0],
])
future = np.triu(np.ones_like(scores, dtype=bool), k=1)

def softmax_rows(matrix):
    exp = np.exp(matrix - matrix.max(axis=-1, keepdims=True))
    return exp / exp.sum(axis=-1, keepdims=True)

unmasked = softmax_rows(scores)
masked = softmax_rows(np.where(future, -np.inf, scores))
print("unmasked future mass:", f"{unmasked[future].sum():.3f}")
print("masked future mass:", f"{masked[future].sum():.3f}")

Output

unmasked future mass: 1.940
masked future mass: 0.000

If that second value isn't zero in a causal decoder, fix the mask before trusting any metric.

Mastery check

Key concepts

Token IDs become positioned vectors with shape [B, T, d_model].
Scaled dot-product attention mixes visible token values using query-key relevance.
A causal mask gives future positions zero attention weight.
Heads run in parallel inside a block; blocks run sequentially.
Attention communicates across positions; the FFN transforms each position.
Residual paths and normalization let same-shaped blocks stack.
In the pre-LN pipeline shown here, a final normalization prepares the residual stream before the output projection turns the last hidden state into next-token logits.

Evaluation rubric

Foundational: Traces Key blocked by from token IDs through hidden states to a next-token distribution.
Foundational: Explains why the causal mask must hide policy from the by position during training.
Intermediate: Derives the score and output shapes for one attention head and for a multi-head merge.
Intermediate: Implements causal attention with a future-mask assertion.
Intermediate: Explains attention, FFN, residual, and normalization roles without treating them as interchangeable.
Advanced: Distinguishes encoder-only, decoder-only, and encoder-decoder visibility rules and selects a variant for a stated task.

Common pitfalls

Treating query, key, or value as a fixed property of a word rather than a learned per-layer projection.
Applying softmax before causal masking, which allows future answers to influence the output.
Confusing parallel attention heads with sequential decoder blocks.
Projecting to vocabulary size inside every block instead of once after the stack.
Assuming every Transformer has the same normalization placement or cross-attention path.

Follow-up questions

Next Step

Continue to Language Modeling & Next Tokens

You can now run the decoder forward path and obtain next-token logits. Next you'll shift token sequences into inputs and targets, then apply the <span data-glossary="cross-entropy">cross-entropy</span> objective introduced earlier at every predicted position.

PreviousAutoencoders and VAEs

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Attention Is All You Need.

Vaswani, A., et al. · 2017

https://arxiv.org/abs/1706.03762

Improving Language Understanding by Generative Pre-Training.

Radford, A., et al. · 2018

https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

Deep Residual Learning for Image Recognition.

He, K., Zhang, X., Ren, S., Sun, J. · 2015 · CVPR 2016

https://arxiv.org/abs/1512.03385

Layer Normalization.

Ba, J. L., Kiros, J. R., & Hinton, G. E. · 2016

https://arxiv.org/abs/1607.06450

On Layer Normalization in the Transformer Architecture.

Xiong, R., et al. · 2020 · ICML 2020

https://arxiv.org/abs/2002.04745

GPT-2 Source Implementation.

OpenAI · 2019

https://github.com/openai/gpt-2/blob/master/src/model.py

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Devlin, J., et al. · 2019 · NAACL 2019

https://arxiv.org/abs/1810.04805

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.

Raffel, C., et al. · 2020 · JMLR

https://arxiv.org/abs/1910.10683

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

The Transformer Architecture End-to-End

Tokens become positioned vectors

Attention lets a token read earlier tokens

Scale before softmax

The causal mask prevents answer leakage

Why can't the representation at `by` attend to `policy` while training a decoder-only next-token model?

Heads run in parallel; blocks run in sequence

A block alternates communication and local transformation

Residual paths and normalization

Implement masked attention in PyTorch

Assemble one decoder block

Project the final position into a vocabulary

Encoder, decoder, and encoder-decoder visibility

Diagnose a broken forward pass

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

For an input tensor with shape `[2, 10, 768]` and 12 heads, what is each head's query shape?

Why is a Transformer decoder block able to stack many times without a shape adapter?

When would cross-attention appear in a Transformer used for deployment-policy translation?

What test catches the most damaging causal-attention implementation error?

Mastery Check

Discussion

Tokens become positioned vectors

Attention lets a token read earlier tokens

Scale before softmax

The causal mask prevents answer leakage

Why can't the representation at by attend to policy while training a decoder-only next-token model?

Why can't the representation at by attend to policy while training a decoder-only next-token model?

Heads run in parallel; blocks run in sequence

A block alternates communication and local transformation

Residual paths and normalization

Implement masked attention in PyTorch

Assemble one decoder block

Project the final position into a vocabulary

Encoder, decoder, and encoder-decoder visibility

Diagnose a broken forward pass

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

For an input tensor with shape [2, 10, 768] and 12 heads, what is each head's query shape?

For an input tensor with shape [2, 10, 768] and 12 heads, what is each head's query shape?

Why is a Transformer decoder block able to stack many times without a shape adapter?

Why is a Transformer decoder block able to stack many times without a shape adapter?

When would cross-attention appear in a Transformer used for deployment-policy translation?

When would cross-attention appear in a Transformer used for deployment-policy translation?

What test catches the most damaging causal-attention implementation error?

What test catches the most damaging causal-attention implementation error?

Mastery Check

Discussion

Why can't the representation at `by` attend to `policy` while training a decoder-only next-token model?

For an input tensor with shape `[2, 10, 768]` and 12 heads, what is each head's query shape?