Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
The last chapter treated SGLang as a serving runtime: tokenize the request, reuse a radix prefix, pack a step, sample tokens. slime sits on the other side of that HTTP boundary during reinforcement-learning (RL) post-training.
A coding-agent rollout edits a sandbox, passes tests in a clean grading environment, and earns reward 1.0. After training on the batch, pass rate falls even though GPUs are busy and every SGLang health check is green. That is the failure scenario we'll investigate, not a measured slime incident.
Don't call that an unstable objective yet. A tool observation may have entered the loss, one trajectory may have been counted twice, or SGLang may still be serving the weights from before Megatron's step.
Follow one receipt through that incident. Call it rollout 7. The prompt asks the model to fix a failing test. The model writes a tool call, the sandbox returns an observation, the model writes a patch, and a clean grader returns reward 1.0. slime has to keep those sampled token IDs, the loss mask, and the group identity intact from SGLang to Megatron.[1][2]
Ray placement determines which resources can overlap, timing policy determines how old a sample may be, and weight synchronization determines when a new policy becomes visible. We'll follow rollout 7 across those boundaries.
Source behavior is pinned to slime commit 681b3adca54105d5ecd3fb822fa0dc58a427e0f9; living documentation was checked September 2, 2026. The examples below execute CPU-only fixtures, not Ray actors, SGLang inference, or Megatron training. Token IDs, rewards, and weight bytes are invented to expose correctness conditions.[1]
Keep Megatron and SGLang, own the handoff
RL post-training is an online loop, not a static dataset pass. A prompt reaches a policy, the policy samples one or more responses, an environment or verifier assigns reward, and a trainer changes the policy. The next prompt should eventually see that new policy, while enough work overlaps to keep GPUs busy. Long responses, tool calls, and failed environments make each iteration irregular.
That irregularity is where separate stacks drift. Math, tool use, multi-turn agents, asynchronous sampling, and supervised fine-tuning can each grow different assumptions about tokens, rewards, and retries. slime keeps the training and serving engines near their upstream interfaces, then puts data-generation and reward logic behind explicit hooks.[2]
The ownership split is deliberate. Megatron owns high-throughput parameter updates. SGLang owns token generation behind a router. Ray owns placement and actor lifecycle. slime owns the handoff, sample contract, timing policy, and checks that keep rollout data aligned with training.
An agent workflow can use a sandbox, search service, browser, tool server, or another model. Its adapter must still preserve the token-level training contract. slime's hook interface doesn't authorize those tools: credentials, network access, sandbox isolation, and write permissions belong to the application running the rollout.
Why does slime expose custom generation instead of defining one agent class?
Answer
Agent environments have different message formats, tools, state, and stopping rules. A custom generation hook lets each workflow keep its own loop while returning token-level Samples that the shared rollout and training code can validate.
A token ledger between two engines
Start with a prediction. If one response contains model tokens, a tool observation, and more model tokens, what tells the trainer which positions deserve credit?
A rollout acts as a ledger entry. Its prompt and sampled response carry the sequence, while log probabilities, loss masks, reward, status, and rollout ID explain how that sequence should affect the next gradient update. Ray decides where the ledger work runs. The Data Buffer keeps entries available until the trainer consumes a complete group.
The policy appears twice, with different jobs. SGLang reads current weights and produces tokens quickly. Megatron reads those tokens and computes losses, gradients, and optimizer updates. A weight-sync operation is the commit that changes what SGLang will sample next. Delay that commit and data becomes off-policy by design, so timing must be recorded rather than inferred later.
Ray places the rollout and training actors; the arrows here carry data or weight updates, not placement requests. A generated response isn't ready for training until its token IDs, response length, loss mask, reward, and acceptance status agree. A weight update isn't complete until rollout engines have loaded the intended version and can serve again.
Now ask where that ledger can live. Which GPUs may generate while others train, and which sync path makes that overlap legal? Ray answers the placement question before SGLang or Megatron sees a token.
Ray placement makes topology a contract
Ray placement groups reserve bundles such as one GPU plus one central processing unit (CPU) slot. slime builds one placement group, labels bundles for actor training or rollout, and sorts them by node and physical GPU so distributed rank order stays stable. This code is infrastructure, but it decides whether a sync path can work.
With --colocate, actor training and rollout share a GPU allocation. One side must offload before the other can use that memory. Without colocation, actor and rollout receive separate bundles and can overlap more freely, at the cost of extra hardware and a weight-transport boundary. Async training requires the decoupled shape in train_async.py; its first assertion rejects colocation.
| Topology | Ray allocation | What it enables | Main pressure |
|---|---|---|---|
| Colocated | One shared GPU range | Less hardware, simple local transfer | Memory eviction and blocked overlap |
| Decoupled | Actor and rollout ranges | Training and generation can overlap | Weight transport and shared storage |
| External rollout | Actor range plus external SGLang endpoints | Separately operated serving cluster | Compatible management APIs, storage, and versions |
The same placement group can describe server groups with different sizes. SGLang configuration supports regular, prefill, decode, and placeholder groups. A placeholder reserves GPU slots without creating an engine, while Ray still accounts for every slot. Heterogeneous layout changes where requests run, not what a Sample means.
slime passes Megatron arguments directly and prefixes installed SGLang arguments with --sglang-. Tensor, pipeline, expert, context, and data parallel settings stay visible to Megatron, while serving flags stay available to SGLang. A run should persist resolved arguments, bundle-to-node mapping, topology, sync mode, and policy version alongside its rollout receipt. The launch file explains intent; that resolved record proves what ran.[3]
When should a reader choose decoupled placement for a long-horizon agent?
Answer
Choose it when generation and training need independent memory or when agent rollouts have a heavy latency tail. Keep a versioned transport and recovery plan because every overlap introduces a weight-staleness and visibility boundary.
SGLang and sgl-router own rollout
At the rollout boundary, ask which representation can be trusted after a tool loop: decoded text or the IDs sampled by the policy?
slime launches SGLang in server mode and keeps a router endpoint in front of its workers. A regular sample is tokenized, sent to /generate with return_logprob=True, and returned with response token IDs, per-token log probabilities, and metadata. Text helps an operator inspect a run; token IDs remain the training source of truth.
The previous chapter's radix prefix cache still matters here. With router_policy=consistent_hashing, slime sends X-SMG-Routing-Key from the sample session ID, so one multi-turn session can return to the same worker and reuse its prefix cache. That improves locality, not provenance. The trajectory manager still has to prove which tokens came from the model.
The default rollout function creates sample groups, runs generation concurrently, computes rewards, and waits until the requested batch is valid. Dynamic sampling can discard groups whose reward variation is unhelpful. Partial rollout mode can abort requests and keep unfinished samples for a later pass. SGLang's /abort_request endpoint supports that early stop when oversampling has already found enough valid responses.[2]
That gives an operator two separate questions. Did the request finish with a valid trajectory? If not, did the abort or retry leave a visible status rather than a partial response that looks complete? Keep those answers beside the sample, not only in server logs.
For mixture-of-experts (MoE) rollouts, the same pass-through lets you set --sglang-moe-a2a-backend deepep. That doesn't change the Sample contract. It changes how expert-parallel tokens move inside SGLang, which is why DeepEP is the next chapter.
There are two useful debug boundaries. --debug-rollout-only exercises generation without training; use --save-debug-rollout-data with a path to retain its samples. --debug-train-only isolates the training side, with --load-debug-rollout-data providing a saved batch for replay. These modes help place a reward or tokenization defect before or after the Data Buffer.[4]
Data Buffer is the handoff contract
The logical Data Buffer is the bridge from prompt initialization to generated samples. A data source supplies groups, the rollout function fills them, and the manager converts Samples into tensors and schedules them across data-parallel ranks. The optional slime_plugins/rollout_buffer runs a separate HTTP service for agent trajectory generation, but it still returns the same grouped evidence.
Each Sample carries more than text. tokens contains prompt IDs followed by the response-side sequence, including any tool observations inserted during the trajectory. response_length counts that suffix, not just model-generated tokens. loss_mask identifies response positions that contribute to policy loss. A zero mask excludes the observation as a training target; it doesn't remove the observation from the context used to predict later model tokens.
Keep three identities separate: group_index groups independent attempts at one prompt, rollout_id groups segments from one agent execution, and weight_versions records serving-policy versions. None substitutes for the others. In particular, the manager's numbered rollout batch isn't the same thing as every Sample's trajectory ID.[5]
Predict before reading the figure. Rollout 7 has prompt IDs [101, 102], then a six-token response: model tool call [11, 12], tool observation [50, 51], and model answer [13, 14]. The prompt is outside the response mask, and only four model positions should train.
| Field | Producer | Training meaning | Failure if missing |
|---|---|---|---|
tokens | SGLang or custom adapter | Exact sequence for forward pass | Re-tokenization changes target |
rollout_log_probs | SGLang | Off-policy correction or diagnostics | Mismatch is invisible |
loss_mask | Custom adapter or converter | Which positions enter target loss | Tool observations get trained |
reward | Verifier or reward model | Advantage input | No learning signal |
rollout_id | Data source or fan-out hook | One trajectory denominator | Segments count as separate rollouts |
status | Rollout worker | Complete, truncate, abort, or fail | Acceptance policy can't distinguish outcomes |
The manager can normalize rewards for estimators such as Group Relative Policy Optimization (GRPO), then prepares training fields and a data-parallel schedule. Custom generation can return several Samples for one compacted trajectory. Those siblings need one rollout_id, so the configured loss reduction can use the combined trainable-token count. Don't assume flattening variable-sized branches preserves the intended prompt-level reward groups: inspect reward postprocessing as well as the loss denominator.
The practical test is simple: sum trainable positions, inspect status, and group by rollout ID before the GPU step. If those counts change when an adapter changes its message format, the bug is in the handoff even if reward calculation still returns a number.
An external rollout buffer adds another boundary. Its generators write items over HTTP, group them by instance ID, and expose valid groups to the trainer. If the service loses a process or clears temporary data, the trainer may still be healthy while its next batch is empty. Treat buffer availability, group completeness, and metadata freshness as separate health signals.
This local validator deliberately accepts only completed trajectories. Slime itself has additional statuses, including usable truncations and recoverable failures; their acceptance policy is workload-dependent. The dataclass below is a fixture, not the upstream Sample class. It checks aligned log probabilities, binary masks, nonempty model output, and finite reward before returning a count:
1from dataclasses import dataclass, replace
2from math import isfinite
3
4@dataclass
5class Sample:
6 tokens: list[int]
7 loss_mask: list[int]
8 response_length: int
9 reward: float
10 rollout_id: int
11 status: str
12 rollout_log_probs: list[float]
13
14def prompt_len(sample: Sample) -> int:
15 return len(sample.tokens) - sample.response_length
16
17def trainable_positions(sample: Sample) -> int:
18 if sample.status != "COMPLETED":
19 raise ValueError("Fixture policy accepts completed trajectories only")
20 if not 0 < sample.response_length < len(sample.tokens):
21 raise ValueError("Require a prompt and a nonempty response suffix")
22 if len(sample.loss_mask) != sample.response_length:
23 raise ValueError("Mask length differs from response length")
24 if any(mask not in (0, 1) for mask in sample.loss_mask) or not any(sample.loss_mask):
25 raise ValueError("Require binary mask and at least one trainable position")
26 if len(sample.rollout_log_probs) != sample.response_length:
27 raise ValueError("Log probabilities must align with response positions")
28 if not isfinite(sample.reward):
29 raise ValueError("Reward must be finite")
30 for mask, log_prob in zip(sample.loss_mask, sample.rollout_log_probs):
31 if not isfinite(log_prob) or log_prob > 0 or (mask == 0 and log_prob != 0):
32 raise ValueError("Invalid sampled log probability or masked placeholder")
33 return sum(sample.loss_mask)
34
35sample = Sample(
36 tokens=[101, 102, 11, 12, 50, 51, 13, 14],
37 loss_mask=[1, 1, 0, 0, 1, 1],
38 response_length=6,
39 reward=1.0,
40 rollout_id=7,
41 status="COMPLETED",
42 rollout_log_probs=[-0.2, -0.3, 0.0, 0.0, -0.4, -0.5],
43)
44assert prompt_len(sample) == 2
45assert trainable_positions(sample) == 4
46
47invalid = [
48 replace(sample, status="ABORTED"),
49 replace(sample, loss_mask=[1, 1]),
50 replace(sample, loss_mask=[1, 2, 0, 0, 1, 1]),
51 replace(sample, loss_mask=[0] * 6),
52 replace(sample, response_length=9),
53 replace(sample, rollout_log_probs=[-0.2]),
54 replace(sample, rollout_log_probs=[0.2, -0.3, 0.0, 0.0, -0.4, -0.5]),
55 replace(sample, rollout_log_probs=[-0.2, -0.3, -0.8, 0.0, -0.4, -0.5]),
56 replace(sample, reward=float("nan")),
57]
58for candidate in invalid:
59 try:
60 trainable_positions(candidate)
61 except ValueError:
62 pass
63 else:
64 raise AssertionError("Invalid fixture accepted")
65print(f"prompt={prompt_len(sample)}, response={sample.response_length}, "
66 f"trainable={trainable_positions(sample)}, rejected={len(invalid)}")1prompt=2, response=6, trainable=4, rejected=9Four response positions train; the two observation positions remain contextual input. The zeros in their log-probability slots are alignment placeholders, not claims that the model sampled those tokens with probability one. A correctly shaped mask still needs provenance checks against the adapter's actual event stream.
Megatron owns the train step
Megatron receives scheduled token tensors from Ray's object store or tensor transport. Its actor workers initialize distributed process groups, create model and optimizer state, compute log probabilities, and run the chosen RL loss. A critic can run alongside the actor. GRPO-style paths can train without one when relative rewards provide the advantage signal.
The actor group exposes async_train, save_model, and update_weights. The name async_train is a Ray dispatch detail, not proof of a fully asynchronous RL policy. train.py still waits for those references before saving or updating rollout weights. Read the caller before inferring timing from a method name.
Megatron parallelism remains native. Tensor, pipeline, expert, context, and data parallel groups are created by Megatron. slime converts gathered parameters to the format SGLang expects through raw conversion or a Megatron Bridge path. That keeps optimizer and checkpoint behavior close to Megatron, but version drift in either upstream engine can surface at the boundary.
Once rollout 7's tensors are on the actor, the remaining question is timing. Does generation wait for this train step, overlap the next batch, or keep a warm worker across boundaries? The answer sets how much policy staleness the receipt must explain.
Three timing models
All three modes use the same Sample contract. They differ in when a request may run relative to an optimizer step and when a new policy version becomes visible. Tie each name to the concrete loop in the repository before comparing throughput.
Synchronous: finish, train, sync
Predict the failure mode first: if one verifier takes ten times longer, what waits? train.py asks the RolloutManager for one rollout, waits for its data, trains on it, and then calls actor_model.update_weights(). If rollout and training are colocated, it can offload memory between those phases.
After a successful sync, the next generation round can use the updated actor. This minimizes cross-round policy lag; it doesn't prove exact on-policy optimization. Multiple optimizer steps, sampling transforms, precision differences, or carried-over partial trajectories still need accounting.
The cost is idle time. A slow verifier or one long response holds the batch open, and training can't use GPUs while generation is still collecting that batch. Dynamic sampling and partial aborts reduce some waste, but the loop remains round-bound.
Async: start next rollout early
Predict the receipt here: a sample can be valid and still come from an older policy. train_async.py starts rollout_manager.generate.remote(next_id) before training the current rollout. Training and generation occupy separate GPU ranges.
A later update_weights_interval controls how often the actor waits for the next rollout future before changing weights. The ordinary async loop drains that future before sync. A completed future isn't a universal proof that every custom background task has stopped; a persistent rollout worker needs its own pause/abort boundary.[6]
Async reduces idle time, but samples can be generated by an older policy. That staleness is a chosen property, not a hidden bug. Compare rollout and actor weight versions in logs. Increase overlap only after reward and Kullback-Leibler (KL) behavior stay within the experiment's contract.
Fully async: keep a warm worker
The fully-async example uses train_async.py plus slime.rollout.fully_async_rollout.generate_rollout_fully_async. A process-wide thread owns an asyncio loop, keeps a fixed number of generate_and_rm_group tasks in flight, and draws new groups from the global data buffer as soon as slots open. Completed groups wait in an output queue until the next training call needs its target batch.
This worker decouples in-flight concurrency from one rollout's batch size. It sorts completed groups by sample index for deterministic handoff, requeues a group containing an aborted Sample, and doesn't support evaluation mode. It also doesn't own pause or weight-update signaling. Each in-flight generation short-circuits on those signals and surfaces ABORTED. Long-tail agents can continue while a later train step consumes completed work, but policy staleness and queue backpressure become operational metrics.
If queue age rises while GPUs look busy, inspect aborted groups and policy versions before raising concurrency. More workers can make the queue look healthy while producing samples too old to use.
| Mode | Generation boundary | Weight update point | Best fit | Main failure surface |
|---|---|---|---|---|
| Sync | Batch must finish before train | Every loop | Debugging and minimal cross-round lag | Longest sample blocks all work |
| Async | Next batch starts during train | Interval, after future drain | Decoupled throughput | Stale policy or update race |
| Fully async | Warm worker spans boundaries | Caller-defined, with queue | Long-tail agent trajectories | Queue growth, abort requeue, no eval |
What makes fully async different from only calling ray.remote() earlier?
Answer
The warm worker keeps generation tasks alive across rollout calls and consumes the shared Data Buffer continuously. A single early remote() overlaps one known future; fully async manages a persistent queue, concurrency cap, completion queue, ordering, and abort requeue.
Weight synchronization is a correctness boundary
The optimizer step changes the actor, but rollout workers may live in separate processes or hosts. The actor needs both a transport for parameters and a lifecycle that prevents generation against a partially loaded update. First resolve the actual branch: full, non-disk updates use local tensor transfer for colocation and distributed NCCL otherwise. Full disk and delta disk are separate alternatives.
With colocation, UpdateWeightFromTensor uses CUDA interprocess communication (IPC) handles rather than the distributed broadcast path. The nccl transport default alone therefore doesn't identify what moved the weights. Delta disk rejects colocation in this snapshot.[7]
Full NCCL
For non-colocated --update-weight-mode full --update-weight-transport nccl, NCCL (the NVIDIA Collective Communications Library) carries the update. Training rank 0 pauses generation and flushes SGLang caches. Pipeline-parallel (PP) source ranks gather tensor-parallel (TP) and expert-parallel (EP) shards into Hugging Face (HF)-shaped chunks, then broadcast them to rollout engines. Engines resume after the chunks and any quantization post-processing finish. The rollout-engine lock serializes broadcast operations; it isn't a durable, cross-system transaction log.
Full NCCL avoids writing a checkpoint for each transfer, but its latency depends on gathering, network topology, and synchronization. A restarted engine needs another transfer or a separate checkpoint; the broadcast itself leaves no replay artifact.
Full disk
Full disk uses update-weight-mode=full and update-weight-transport=disk. Each sync writes a canonical Hugging Face checkpoint directory under a version such as weight_v000003. The rollout engine pulls that directory, optionally into a host-local checkpoint, and reloads through its ordinary update_weights_from_disk endpoint. A post-write hook can publish files to an object-store-backed mount before hosts read them.
A versioned directory makes the update inspectable, but not automatically durable. Full-disk reload normally removes the published directory unless --update-weight-disk-keep-files is enabled. Retention and durable storage still need explicit configuration. The file is a weight export, not a complete optimizer-and-data-source recovery checkpoint.
Publication and an optional host-local pull occur before generation is paused. Pause, cache flush, and reload form the live-weight transition. A successful trainer-side write isn't proof that every engine can read the same bytes, and a successful reload on one engine isn't a fleet-wide readiness check.
Delta disk
Delta mode is disk-only and non-colocated. The trainer first captures a CPU baseline seeded from --hf-checkpoint, then diffs each gathered HF tensor on later syncs. Changed bytes are compressed with zstd and written as a self-describing version. Each rollout host applies the delta into its local full checkpoint through /pull_weights, verifies per-tensor checksums, and reloads through the same disk endpoint used by full sync.
Exclusive-or (XOR) encodes byte differences; compression savings depend on how much changes, not on the mode name. A raw XOR patch must be applied once to the intended base. overwrite stores changed positions and values, so repeating the same writes doesn't toggle them back, but it still needs the correct base and version ordering. The index records those identities and a new-state checksum. Neither an xxHash checksum nor a version number authenticates an untrusted publisher.[8]
1def xor_bytes(left: bytes, right: bytes) -> bytes:
2 if len(left) != len(right):
3 raise ValueError("Patch and base lengths must match")
4 return bytes(x ^ y for x, y in zip(left, right))
5
6base = bytes([1, 2, 3, 4])
7new = bytes([1, 2, 8, 4])
8delta = xor_bytes(new, base)
9applied_once = xor_bytes(base, delta)
10applied_twice = xor_bytes(applied_once, delta)
11assert applied_once == new
12assert applied_twice == base
13try:
14 xor_bytes(base, delta[:-1])
15except ValueError:
16 pass
17else:
18 raise AssertionError("Truncated delta accepted")
19print(list(delta), list(applied_once), list(applied_twice))1[0, 0, 11, 0] [1, 2, 8, 4] [1, 2, 3, 4]Byte 8 ^ 3 is 11. Applying the patch once reaches the new state. Applying it twice restores the base, which is why a repeated XOR is dangerous: it can look successful while serving the old model.
Keep source version, declared base, destination, checksum, reload result, and served version together. If a retry can't establish whether the local base was already patched, stop new admissions and repair from a known checkpoint. Don't blindly XOR again. Crash-safe recovery also needs atomic persistence of bytes and version state; the four-byte example doesn't implement that protocol.
These are argument fragments, not a runnable shell command. Add them to a compatible non-colocated training launch with model, dataset, actor, and SGLang configuration. The shared directory must be visible to all participating hosts; the local checkpoint path is host-local:
1--update-weight-mode delta \
2--update-weight-transport disk \
3--update-weight-disk-dir /shared/fs/delta-updates \
4--update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt \
5--update-weight-delta-encoding xor \
6--update-weight-delta-checksum xxh3-128
| Path | Carrier | Engine-side state | Strong guard | What it can't hide |
|---|---|---|---|---|
| Full colocated | CUDA IPC tensor handles | Shared-device weight transfer | Offload and engine lifecycle | IPC isn't a durable checkpoint |
| Full NCCL | NCCL broadcasts | Live SGLang weights | Pause and flush around chunks | A lost engine needs another transfer |
| Full disk | Shared filesystem | Complete HF weight export | Versioned directory and reload | Visibility and retention are separate |
| Delta disk | Shared filesystem plus local checkpoint | Patched full checkpoint | Base version and per-tensor checksum | A bad baseline poisons every later diff |
Agentic customization keeps token provenance
Start agentic work with --custom-generate-function-path when one prompt should run a custom loop. The function can call tools, retrieval, a browser, a sandbox, or another service, then return one Sample or a list of sibling Samples. Use --custom-rm-path for verifier or environment reward.[9][10]
Replace the whole rollout function only when scheduling or buffering can't fit the default per-sample loop. Keeping the default orchestration leaves grouping, validation, and handoff behavior visible while the custom hook owns the environment-specific work.
Calling a tool is straightforward; preserving token provenance around that call takes care. Model-generated output tokens should have loss_mask=1. Tool observations, template fragments, and environment text inserted into the response suffix should normally have loss_mask=0; the initial prompt sits outside that mask. The adapter records sampled token IDs and log probabilities directly from SGLang instead of decoding text and tokenizing it again.
When a reward changes, inspect this boundary before changing the objective. Compare the sampled IDs, mask, response length, verifier input, and policy version. A readable transcript can hide a different training target.
slime ships protocol adapters for that job. AnthropicAdapter and OpenAIAdapter render a chat template, call SGLang with input_ids and return_logprob=True, and export the returned IDs as trainable segments. They also pass a stable session_id as X-SMG-Routing-Key. The adapters are a convenience layer, not a second agent framework.
One agent execution can fan out into subagent or compaction segments. Return them with the same rollout_id, but don't mechanically divide every reward by the segment count. The customization guide describes reward / K as one pattern; whether it's appropriate depends on reward-to-advantage processing and loss reduction. In the pinned per-rollout-mean path, sibling losses already share the rollout's total trainable-token denominator. Scaling their advantages down again would reduce that rollout's contribution.[9][11]
Hold per-token loss terms fixed at [1, 3, 5, 7]. Splitting them into two segments must not change their mean from 4 to 8, or halve it to 2. This isolates the reduction arithmetic; it doesn't simulate GRPO reward normalization or gradients:
1segments = [[1.0, 3.0], [5.0, 7.0]]
2denominator = sum(len(segment) for segment in segments)
3one_rollout = sum(sum(segment) / denominator for segment in segments)
4wrong_independent = sum(sum(segment) / len(segment) for segment in segments)
5wrong_extra_halving = sum(sum(segment) / 2 / denominator for segment in segments)
6assert one_rollout == 4.0
7assert wrong_independent == 8.0
8assert wrong_extra_halving == 2.0
9# A different split preserves the same aggregate denominator.
10assert sum(sum(s) / denominator for s in [[1.0], [3.0, 5.0, 7.0]]) == one_rollout
11print(f"one rollout={one_rollout}; independent sum={wrong_independent}; "
12 f"extra halving={wrong_extra_halving}")1one rollout=4.0; independent sum=8.0; extra halving=2.0If segments land in different microbatches, compute the denominator over the whole training step first. A microbatch-local denominator would count one rollout repeatedly. Per-token loss modes or custom objectives have different normalization rules, so check the configured reducer before deciding how to assign segment rewards.
| Workflow | Starting hook | Evidence to preserve | Useful artifact |
|---|---|---|---|
| Search or retrieval-augmented generation (RAG) | custom_generate | Query, retrieved context, sampled tokens | Search trace plus reward report |
| Tool agent | custom_generate + custom_rm | Tool calls, observations, verifier result | Session trajectory and test log |
| Coding agent | Adapter or custom generate | Model tokens, sandbox diff, clean tests | Patch, rollout dump, grader output |
| Multi-agent | rollout_function or fan-out generate | Branch IDs and shared rollout ID | Per-branch masks and explicit reward/reduction policy |
| Long-tail agent | Fully-async rollout function | Queue age, abort status, weight version | Replayable debug dump |
The coding-agent example makes this concrete. A test runner edits a fresh sandbox, captures a diff, and grades that diff in a second clean sandbox. The training target is still the model's token stream, not final text copied from a log. This split lets teams change the runner without changing Megatron's loss code.
1from slime.utils.types import Sample
2
3async def custom_generate(args, sample, sampling_params):
4 trajectory = await run_agent(sample.prompt, sampling_params=sampling_params)
5 sample.tokens = trajectory.prompt_ids + trajectory.response_tokens
6 sample.loss_mask = trajectory.response_loss_mask
7 sample.response_length = len(trajectory.response_tokens)
8 sample.rollout_log_probs = trajectory.response_log_probs
9 sample.reward = await verify(trajectory)
10 if sample.rollout_id is None:
11 sample.rollout_id = sample.index
12 if sample.rollout_id is None:
13 raise ValueError("A trajectory ID is required")
14 sample.status = Sample.Status.COMPLETED
15 return sampleThis is an integration sketch: run_agent, verify, the trajectory type, and tool permissions are application-owned and omitted, so it isn't a drop-in script. Use the real status enum, and test is None when choosing an identity because rollout_id=0 is valid. Mark completion only after the chosen terminal condition and verifier succeed; failures need an explicit retry/drop policy.
Applications and ecosystem
The pinned README identifies slime as infrastructure for the GLM family and includes launch examples for other model families. That is project-reported adoption, not a controlled performance comparison. Read the model-specific launch script and report before carrying a memory, throughput, or quality claim into a different configuration.[1]
Choose a starting example by the boundary your workload stresses: verifier quality, multi-turn context, tool latency, or weight transport. A working math-verifier run doesn't establish that a coding sandbox has safe permissions or that a long-lived agent handles aborted generations correctly.
| Application pressure | slime surface | Question to measure |
|---|---|---|
| Verifiable answers | Reward model and group filter | Are reward groups informative? |
| Long context or tools | SGLang router and session key | Are requests pinned without hot spots? |
| Slow sandboxes | Fully-async worker or partial rollout | Is queue age bounded? |
| Separate serving cluster | Disk or delta weight sync | Can every host prove its base version? |
| MoE rollout communication | --sglang-moe-a2a-backend deepep | Did expert-parallel tokens take the intended path? |
| New model family | Native Megatron and SGLang pass-through | Which conversion and parser contracts changed? |
Treat each application as an experiment with a receipt. Save resolved launch arguments, model checkpoint, data revision, reward code, sync mode, topology, policy versions, rollout dumps, and metrics. Without that receipt, a green reward curve can't tell you whether the policy improved or the verifier changed.
Strengths, limits, and failure paths
slime's strengths come from narrow ownership boundaries. It doesn't hide Megatron parallelism behind a new trainer abstraction or flatten SGLang's serving flags to a lowest common denominator. Upstream engine work can land without a second trainer API, while RL-specific sample and synchronization rules stay in one place.[2]
| Strength | Why it helps | Cost or limit |
|---|---|---|
| Native Megatron path | Mature distributed training and checkpoint tools remain available | Megatron version changes can break bridges |
| SGLang-native rollout | Server, router, caching, and parser features stay visible | SGLang is the chosen rollout backend |
| Ray placement and actors | One resource vocabulary for train and rollout | Ray scheduling and object-store state need operations |
| Explicit Sample contract | Token provenance and group identity are inspectable | Custom hooks must honor many fields |
| Multiple sync modes | Topology can choose NCCL, full disk, or delta disk | Each mode has different recovery and storage risks |
| Lightweight core | Teams can add environments without a framework fork | Application policy, auth, and governance stay outside |
Most production failures cross a boundary. A rollout server can stay alive while serving old weights. A reward function can return a number while masking every response token. A fully-async queue can stay busy while repeatedly requeueing aborted groups. Name the boundary first, then inspect the receipt that crosses it.
| Symptom | Likely boundary | Check first | Guardrail |
|---|---|---|---|
| Reward changes with no code change | Data or verifier | Dataset revision and reward config | Persist input and reward metadata |
| KL spikes after sync | Weight transport | Engine version, pause/flush logs, checksum | Compare weights before serving |
| Agent learns tool observations | Token trajectory | loss_mask around tool messages | Assert mask length and source |
| Fully-async throughput falls | Queue or abort path | Queue age, aborted count, worker logs | Requeue intentionally and cap concurrency |
| Disk sync succeeds on trainer only | Filesystem visibility | Host-local checkpoint and hook output | Publish then pull on every host |
| Job hangs after rollout crash | Fault handling | Health monitor and restart logs | Enable health checks and save replay dumps |
Fault tolerance and reproducibility
--use-fault-tolerance starts heartbeat checks against SGLang servers. If a heartbeat times out, slime stops the unhealthy engine. After the current rollout round finishes, it restarts the engine and applies the correct parameters before future requests. This is rollout-engine recovery, not a promise that a failed trainer rank or preempted cluster job can resume from memory.[4]
Debug replay narrows the search. Save a rollout with --save-debug-rollout-data, load it with --load-debug-rollout-data, and use --debug-train-only to replay conversion and training without starting SGLang. Pair the dump with checkpoints, trace spans, and a pinned launch command. Load only trusted dumps: the pinned replay path uses PyTorch deserialization with weights_only=False, not a safe parser for arbitrary uploads.
Recovery has its own false positives. Health checks can mistake first-run kernel compilation for a dead server, so the docs expose a first-wait setting. Fully-async workers can hide slow samples in a warm queue, so log queue length and completion age. Disk sync can hide stale mounts, so verify every host's local checkpoint and weight version.
The repository's continuous integration (CI) mirrors this split. CPU tests cover Sample behavior, rollout validation, argument contracts, and customization hooks. GPU end-to-end tests cover Megatron, SGLang deployment, async rollout, checkpointing, precision, and replay. A passing unit test doesn't prove a multi-node sync is visible, so keep a small environment-specific smoke run in the release receipt.
Project identity
slime is published through the THUDM organization, whose official profile identifies its THUKEG and Z.ai lineage.[12] The repository citation names Zilin Zhu, Chengxing Xie, Xin Lv, and slime contributors. The introductory blog frames the project around an SGLang-native rollout path, Megatron training, Ray resource management, and custom data generation.[1][2]
| Field | Current project fact |
|---|---|
| Origin | THUDM and Z.ai built slime for post-training workflows that connect Megatron, SGLang, and Ray.[1][2] |
| Founding contributors | The repository citation names Zilin Zhu, Chengxing Xie, Xin Lv, and slime contributors.[1] |
| Stewardship | Z.ai leads the roadmap. Public contribution scope emphasizes bug fixes and general RL optimizations that its CI can verify.[13] |
| Source license | slime source is Apache-2.0, with Zhipu AI copyright notices.[14] |
| Commercial boundary | The framework is open source, but project policy prioritizes Z.ai's internal development roadmap. This is vendor-led governance, not a neutral foundation model.[13] |
| Asset boundary | GLM checkpoints, other model weights, datasets, environments, and reward services retain separate licenses and terms. |
The first release notes describe v0.1.0 as focusing on MoE inference, memory offload, faster parameter updates, Megatron parallel strategies, and strict correctness checks. Treat those statements as release context, not a promise that every current branch has the same performance or feature set.[1]
Use the pinned source for implementation behavior and the LMSYS design post for architectural motivation. Model reports and related systems papers answer different questions; a model's benchmark result doesn't validate every framework path.
Contribution policy is intentionally focused. Bug fixes and general RL optimizations that can be verified through CI are welcome. Large refactors, universal agent abstractions, and changes that can't be tested against routine training stay outside core scope. That governance keeps internal and open development aligned while leaving application-specific systems in their own repositories.[13]
That history explains the project's shape. slime doesn't own every environment, reward model, or serving backend. It concentrates on the handoff between training and rollout, then lets teams compose the rest around stable contracts.
A source and code reading route
This walkthrough uses the official slime repository at commit 681b3adca54105d5ecd3fb822fa0dc58a427e0f9.[1] Read one vertical path before opening every module. Start with the README architecture section and introductory blog to see why Ray, Megatron, SGLang, and the Data Buffer are separate. Then open train.py and train_async.py side by side. Mark each ray.get, generate.remote, and update_weights call. That is the timing model in executable form.
Next inspect slime/ray/placement_group.py and slime/ray/rollout.py. Follow how bundles become server groups, how routers start, and how RolloutManager.generate() converts nested Samples into train data. Read slime/rollout/sglang_rollout.py for prompt IDs, session routing, aborts, and reward calls. Compare slime/rollout/fully_async_rollout.py with the fully-async example to see queue lifetime and abort requeue.
For the weight boundary, read slime/ray/actor_group.py, then the transport modules under slime/backends/megatron_utils/update_weight/: update_weight_from_distributed.py, update_weight_from_disk.py, and update_weight_from_disk_delta.py. Start with actor-group lifecycle. The next two files show transport. The last shows baseline, delta, checksum, and apply ordering. Keep a paper notebook with source version, base version, and engine state for one hypothetical sync.
For agents, read the published customization and agent guides alongside examples/coding_agent_rl. Compare their examples with the pinned Sample type and get_sum_of_sample_mean reducer before adapting a reward rule. For operations, use the fault-tolerance guide to choose a recovery experiment, then inspect the corresponding implementation rather than treating a health check as proof of recovery.[9][10][4]
Carry one question into every repository: which state is authoritative at this step? In slime, authority sits in token IDs during rollout, grouped Samples in the buffer, Megatron parameters during training, and a versioned sync artifact before SGLang serves again.
What to remember
- Ray placement groups make train and rollout topology explicit, including colocation, decoupling, and external engines.
- SGLang and sgl-router generate token evidence; Megatron consumes that evidence for distributed actor training.
- The Data Buffer contract is token based. Preserve log probabilities, loss masks, rewards, statuses, and rollout IDs.
- Synchronous training waits for each round. Async training overlaps one future rollout. Fully async keeps a warm worker and queue across boundaries.
- Colocated full sync uses CUDA IPC. Non-colocated NCCL streams chunks; disk paths publish full weights or checked deltas before live reload.
- Agent hooks can add tools, sandboxes, search, and branches without changing the training kernel, but token provenance remains the correctness rule.
- Fault tolerance restarts rollout engines and supports replay; it doesn't replace cluster-level checkpoint and scheduler recovery.
- Read code by following one Sample and one weight version from source to engine, then verify claims against first-party docs.
Evaluation rubric
- Trace one trajectory's token IDs, response mask, log probabilities, status, and serving-policy versions through rollout and training.
- Distinguish independent attempts from segments of one execution, and calculate the correct loss denominator across microbatches.
- Explain what ordinary async waits for and what a persistent custom rollout worker must coordinate separately.
- Choose a weight-sync path for a stated topology and identify its base-version, visibility, retention, and recovery checks.
Follow-up questions
An agent adapter changes from two segments to five without changing its generated tokens. Which checks establish that its training contribution stayed the same?
Answer
Compare exact token IDs, masks, rollout identity, reward-to-advantage processing, and the denominator across the whole training step. Hold per-token loss terms fixed and confirm that resegmenting preserves their aggregate. Matching final reward alone doesn't establish equal training weight.
A delta reload succeeds on seven of eight engines. What evidence would you require before admitting new requests?
Answer
Check the serving membership and every admitted engine's loaded version. Quarantine the failed engine rather than treating seven reloads as fleet success. Establish its local base and patch receipt; if those are ambiguous, restore known complete weights before rejoining. A healthy endpoint or correct trainer checkpoint alone is insufficient.