Deep Dive - slime

The last chapter treated SGLang as a serving runtime: tokenize the request, reuse a radix prefix, pack a step, sample tokens. slime sits on the other side of that HTTP boundary during reinforcement-learning (RL) post-training.

A coding-agent rollout edits a sandbox, passes tests in a clean grading environment, and earns reward 1.0. After training on the batch, pass rate falls even though GPUs are busy and every SGLang health check is green. Treat this as a diagnostic scenario: inspect training data, token alignment, and serving-weight versions before assigning a cause.

Don't call that an unstable objective yet. A tool observation may have entered the loss, one trajectory may have been counted twice, or SGLang may still be serving the weights from before Megatron's step.

Follow one receipt through that incident. Call it rollout 7. The prompt asks the model to fix a failing test. The model writes a tool call, the sandbox returns an observation, the model writes a patch, and a clean grader returns reward 1.0. slime has to keep those sampled token IDs, the loss mask, and the group identity intact from SGLang to Megatron.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

Three boundaries can hide the drop. Ray placement decides which resources may overlap. Timing policy decides how old a sample may be. Weight synchronization decides when a new policy becomes visible. Rollout 7 has to stay identifiable across all three.

Source behavior is pinned to slime commit 681b3adca54105d5ecd3fb822fa0dc58a427e0f9; the living agent and recovery guides were rechecked October 10, 2026. The recovery section separates the newer retained-serving behavior from this historical source walkthrough. All examples execute CPU-only fixtures with invented token IDs, rewards, and weight bytes. They expose correctness conditions while leaving Ray actors, SGLang inference, and Megatron training for a cluster run.^{[3]Reference 3slime Source Snapshot (681b3ad)https://github.com/THUDM/slime/tree/681b3adca54105d5ecd3fb822fa0dc58a427e0f9}

Keep Megatron and SGLang, own the handoff

RL post-training is an online loop, not a static dataset pass. A prompt reaches a policy, the policy samples one or more responses, an environment or verifier assigns reward, and a trainer changes the policy. The next prompt should eventually see that new policy, while enough work overlaps to keep GPUs busy. Long responses, tool calls, and failed environments make each iteration irregular.

That irregularity is where separate stacks drift. Math, tool use, multi-turn agents, asynchronous sampling, and supervised fine-tuning can each grow different assumptions about tokens, rewards, and retries. slime keeps the training and serving engines near their upstream interfaces, then puts data-generation and reward logic behind explicit hooks.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

The ownership split is deliberate. Megatron owns high-throughput parameter updates. SGLang owns token generation behind a router. Ray owns placement and actor lifecycle. slime owns the handoff, sample contract, timing policy, and checks that keep rollout data aligned with training.

An agent workflow can use a sandbox, search service, browser, tool server, or another model. Its adapter must still preserve the token-level training contract. slime's hook interface doesn't authorize those tools: credentials, network access, sandbox isolation, and write permissions belong to the application running the rollout.

A token ledger between two engines

Start with a prediction. If one response contains model tokens, a tool observation, and more model tokens, what tells the trainer which positions deserve credit?

A rollout acts as a ledger entry. Its prompt and sampled response carry the sequence, while log probabilities, loss masks, reward, status, and rollout ID explain how that sequence should affect the next gradient update. Ray decides where the ledger work runs. The Data Buffer keeps entries available until the trainer consumes a complete group.

The policy appears twice, with different jobs. SGLang reads current weights and produces tokens quickly. Megatron reads those tokens and computes losses, gradients, and optimizer updates. A weight-sync operation is the commit that changes what SGLang will sample next. Delay that commit and data becomes off-policy by design, so timing must be recorded rather than inferred later.

View larger

Ray places the rollout and training actors; the arrows here carry data or weight updates, not placement requests. A generated response isn't ready for training until its token IDs, response length, loss mask, reward, and acceptance status agree. A weight update isn't complete until rollout engines have loaded the intended version and can serve again.

Now ask where that ledger can live. Which GPUs may generate while others train, and which sync path makes that overlap legal? Ray answers the placement question before SGLang or Megatron sees a token.

Ray placement makes topology a contract

Ray placement groups reserve bundles such as one GPU plus one central processing unit (CPU) slot. slime builds one placement group, labels bundles for actor training or rollout, and sorts them by node and physical GPU so distributed rank order stays stable. This code is infrastructure, but it decides whether a sync path can work.

With --colocate, actor training and rollout share a GPU allocation. One side must offload before the other can use that memory. Without colocation, actor and rollout receive separate bundles and can overlap more freely, at the cost of extra hardware and a weight-transport boundary. Async training requires the decoupled shape in train_async.py; its first assertion rejects colocation.

The same placement group can describe server groups with different sizes. SGLang configuration supports regular, prefill, decode, and placeholder groups. A placeholder reserves GPU slots without creating an engine, while Ray still accounts for every slot. Heterogeneous layout changes where requests run, not what a Sample means.

slime passes Megatron arguments directly and prefixes installed SGLang arguments with --sglang-. Tensor, pipeline, expert, context, and data parallel settings stay visible to Megatron, while serving flags stay available to SGLang. A run should persist resolved arguments, bundle-to-node mapping, topology, sync mode, and policy version alongside its rollout receipt. The launch file explains intent; that resolved record proves what ran.^{[4]Reference 4slime Documentationhttps://thudm.github.io/slime/}

SGLang and sgl-router own rollout

At the rollout boundary, ask which representation can be trusted after a tool loop: decoded text or the IDs sampled by the policy?

slime launches SGLang in server mode and keeps a router endpoint in front of its workers. A regular sample is tokenized, sent to /generate with return_logprob=True, and returned with response token IDs, per-token log probabilities, and metadata. Text helps an operator inspect a run; token IDs remain the training source of truth.

The previous chapter's radix prefix cache still matters here. With router_policy=consistent_hashing, slime sends X-SMG-Routing-Key from the sample session ID, so one multi-turn session can return to the same worker and reuse its prefix cache. That improves locality, not provenance. The trajectory manager still has to prove which tokens came from the model.

The default rollout function creates sample groups, runs generation concurrently, computes rewards, and waits until the requested batch is valid. Dynamic sampling can discard groups whose reward variation is unhelpful. Partial rollout mode can abort requests and keep unfinished samples for a later pass. SGLang's /abort_request endpoint supports that early stop when oversampling has already found enough valid responses.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

That gives an operator two separate questions: did the request finish with a valid trajectory, and what status did an abort or retry leave? Record both beside the sample so a partial response cannot look complete.

For mixture-of-experts (MoE) rollouts, the same pass-through lets you set --sglang-moe-a2a-backend deepep. That flag changes how expert-parallel tokens move inside SGLang. It doesn't change what a Sample must contain when it reaches the buffer.

There are two useful debug boundaries. --debug-rollout-only exercises generation without training; use --save-debug-rollout-data with a path to retain its samples. --debug-train-only isolates the training side, with --load-debug-rollout-data providing a saved batch for replay. These modes help place a reward or tokenization defect before or after the Data Buffer.^{[5]Reference 5slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

Data Buffer is the handoff contract

The logical Data Buffer is the bridge from prompt initialization to generated samples. A data source supplies groups, the rollout function fills them, and the manager converts Samples into tensors and schedules them across data-parallel ranks. The optional slime_plugins/rollout_buffer runs a separate HTTP service for agent trajectory generation, but it still returns the same grouped evidence.

Each Sample carries more than text. tokens contains prompt IDs followed by the response-side sequence, including any tool observations inserted during the trajectory. response_length counts the entire response-side suffix, including tool observations. loss_mask identifies response positions that contribute to policy loss. A zero mask excludes the observation as a training target; the observation remains in the context used to predict later model tokens.

Keep three identities separate: group_index groups independent attempts at one prompt, rollout_id groups segments from one agent execution, and weight_versions records serving-policy versions. None substitutes for the others. In particular, the manager's numbered rollout batch isn't the same thing as every Sample's trajectory ID.^{[6]Reference 6Slime Sample Contract (681b3ad)https://github.com/THUDM/slime/blob/681b3adca54105d5ecd3fb822fa0dc58a427e0f9/slime/utils/types.py}

Predict before reading the figure. Rollout 7 has prompt IDs [101, 102], then a six-token response: model tool call [11, 12], tool observation [50, 51], and model answer [13, 14]. The prompt is outside the response mask, and only four model positions should train.

View larger

The manager can normalize rewards for estimators such as Group Relative Policy Optimization (GRPO), then prepares training fields and a data-parallel schedule. Custom generation can return several Samples for one compacted trajectory. Those siblings need one rollout_id, so the configured loss reduction can use the combined trainable-token count. Don't assume flattening variable-sized branches preserves the intended prompt-level reward groups: inspect reward postprocessing as well as the loss denominator.

Check the handoff before the GPU step. Sum trainable positions, inspect status, and group by rollout ID. If those counts change when an adapter changes its message format, the bug is in the handoff even if reward calculation still returns a number.

An external rollout buffer adds another boundary. Its generators write items over HTTP, group them by instance ID, and expose valid groups to the trainer. If the service loses a process or clears temporary data, the trainer may still be healthy while its next batch is empty. Treat buffer availability, group completeness, and metadata freshness as separate health signals.

This local validator deliberately accepts only completed trajectories. Slime itself has additional statuses, including usable truncations and recoverable failures; their acceptance policy is workload-dependent. The dataclass below is a fixture, not the upstream Sample class. It checks aligned log probabilities, binary masks, nonempty model output, and finite reward before returning a count:

from dataclasses import dataclass, replace
from math import isfinite

@dataclass
class Sample:
    tokens: list[int]
    loss_mask: list[int]
    response_length: int
    reward: float
    rollout_id: int
    status: str
    rollout_log_probs: list[float]

def prompt_len(sample: Sample) -> int:
    return len(sample.tokens) - sample.response_length

def trainable_positions(sample: Sample) -> int:
    if sample.status != "COMPLETED":
        raise ValueError("Fixture policy accepts completed trajectories only")
    if not 0 < sample.response_length < len(sample.tokens):
        raise ValueError("Require a prompt and a nonempty response suffix")
    if len(sample.loss_mask) != sample.response_length:
        raise ValueError("Mask length differs from response length")
    if any(mask not in (0, 1) for mask in sample.loss_mask) or not any(sample.loss_mask):
        raise ValueError("Require binary mask and at least one trainable position")
    if len(sample.rollout_log_probs) != sample.response_length:
        raise ValueError("Log probabilities must align with response positions")
    if not isfinite(sample.reward):
        raise ValueError("Reward must be finite")
    for mask, log_prob in zip(sample.loss_mask, sample.rollout_log_probs):
        if not isfinite(log_prob) or log_prob > 0 or (mask == 0 and log_prob != 0):
            raise ValueError("Invalid sampled log probability or masked placeholder")
    return sum(sample.loss_mask)

sample = Sample(
    tokens=[101, 102, 11, 12, 50, 51, 13, 14],
    loss_mask=[1, 1, 0, 0, 1, 1],
    response_length=6,
    reward=1.0,
    rollout_id=7,
    status="COMPLETED",
    rollout_log_probs=[-0.2, -0.3, 0.0, 0.0, -0.4, -0.5],
)
assert prompt_len(sample) == 2
assert trainable_positions(sample) == 4

invalid = [
    replace(sample, status="ABORTED"),
    replace(sample, loss_mask=[1, 1]),
    replace(sample, loss_mask=[1, 2, 0, 0, 1, 1]),
    replace(sample, loss_mask=[0] * 6),
    replace(sample, response_length=9),
    replace(sample, rollout_log_probs=[-0.2]),
    replace(sample, rollout_log_probs=[0.2, -0.3, 0.0, 0.0, -0.4, -0.5]),
    replace(sample, rollout_log_probs=[-0.2, -0.3, -0.8, 0.0, -0.4, -0.5]),
    replace(sample, reward=float("nan")),
]
for candidate in invalid:
    try:
        trainable_positions(candidate)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid fixture accepted")
print(f"prompt={prompt_len(sample)}, response={sample.response_length}, "
      f"trainable={trainable_positions(sample)}, rejected={len(invalid)}")

prompt=2, response=6, trainable=4, rejected=9

Four response positions train; the two observation positions remain contextual input. The zeros in their log-probability slots are alignment placeholders; check the adapter's event stream for sampled-token probabilities and mask provenance.

Megatron owns the train step

Megatron receives scheduled token tensors from Ray's object store or tensor transport. Its actor workers initialize distributed process groups, create model and optimizer state, compute log probabilities, and run the chosen RL loss. A critic can run alongside the actor. GRPO-style paths can train without one when relative rewards provide the advantage signal.

The actor group exposes async_train, save_model, and update_weights. async_train is the Ray dispatch name here. train.py waits for those references before saving or updating rollout weights, so inspect the caller to understand execution timing.

Megatron parallelism remains native. Tensor, pipeline, expert, context, and data parallel groups are created by Megatron. slime converts gathered parameters to the format SGLang expects through raw conversion or a Megatron Bridge path. That keeps optimizer and checkpoint behavior close to Megatron, but version drift in either upstream engine can surface at the boundary.

Once rollout 7's tensors are on the actor, the remaining question is timing. Does generation wait for this train step, overlap the next batch, or keep a warm worker across boundaries? The answer sets how much policy staleness the receipt must explain.

Three timing models

All three modes use the same Sample contract. They differ in when a request may run relative to an optimizer step and when a new policy version becomes visible. Tie each name to the concrete loop in the repository before comparing throughput.

Synchronous: finish, train, sync

Predict the failure mode first: if one verifier takes ten times longer, what waits? train.py asks the RolloutManager for one rollout, waits for its data, trains on it, and then calls actor_model.update_weights(). If rollout and training are colocated, it can offload memory between those phases.

After a successful sync, the next generation round can use the updated actor. This minimizes cross-round policy lag. Account for multiple optimizer steps, sampling transforms, precision differences, and carried-over partial trajectories when estimating policy lag.

The cost is idle time. A slow verifier or one long response holds the batch open, and training can't use GPUs while generation is still collecting that batch. Dynamic sampling and partial aborts reduce some waste, but the loop remains round-bound.

Async: start next rollout early

Predict the receipt here: a sample can be valid and still come from an older policy. train_async.py starts rollout_manager.generate.remote(next_id) before training the current rollout. Training and generation occupy separate GPU ranges.

A later update_weights_interval controls how often the actor waits for the next rollout future before changing weights. The ordinary async loop drains that future before sync. A completed future marks the awaited rollout operation; a persistent rollout worker needs its own pause/abort boundary.^{[7]Reference 7Slime Asynchronous Training Loop (681b3ad)https://github.com/THUDM/slime/blob/681b3adca54105d5ecd3fb822fa0dc58a427e0f9/train_async.py}

Async reduces idle time, but samples can be generated by an older policy. That staleness is a chosen property, not a hidden bug. Compare rollout and actor weight versions in logs. Increase overlap only after reward and Kullback-Leibler (KL) behavior stay within the experiment's contract.

Fully async: keep a warm worker

The fully-async example uses train_async.py plus slime.rollout.fully_async_rollout.generate_rollout_fully_async. A process-wide thread owns an asyncio loop, keeps a fixed number of generate_and_rm_group tasks in flight, and draws new groups from the global data buffer as soon as slots open. Completed groups wait in an output queue until the next training call needs its target batch.

This worker decouples in-flight concurrency from one rollout's batch size. It sorts completed groups by sample index for deterministic handoff, requeues a group containing an aborted Sample, and doesn't support evaluation mode. It also doesn't own pause or weight-update signaling. Each in-flight generation short-circuits on those signals and surfaces ABORTED. Long-tail agents can continue while a later train step consumes completed work, but policy staleness and queue backpressure become operational metrics.

The completion queue is unbounded so a full queue can't block its callback and freeze the asyncio loop. Backpressure instead stops launching new groups once the queued-group count reaches the worker concurrency. Already-running groups can still finish, so this threshold isn't a hard queue-size limit. For concurrency four, four queued groups stop new launches, but an older in-flight group can add a fifth. Training must drain those groups or filter stale data; increasing concurrency doesn't solve a slow consumer.^{[3]Reference 3slime Source Snapshot (681b3ad)https://github.com/THUDM/slime/tree/681b3adca54105d5ecd3fb822fa0dc58a427e0f9}

If queue age rises while GPUs look busy, inspect aborted groups and policy versions before raising concurrency. More workers can make the queue look healthy while producing samples too old to use.

Weight synchronization is a correctness boundary

The optimizer step changes the actor, but rollout workers may live in separate processes or hosts. The actor needs both a transport for parameters and a lifecycle that prevents generation against a partially loaded update. First resolve the actual branch: full, non-disk updates use local tensor transfer for colocation and distributed NCCL otherwise. Full disk and delta disk are separate alternatives.

With colocation, UpdateWeightFromTensor uses CUDA interprocess communication (IPC) handles rather than the distributed broadcast path. The nccl transport default alone therefore doesn't identify what moved the weights. Delta disk rejects colocation in this snapshot.^{[8]Reference 8Slime Megatron Actor and Weight-Sync Selection (681b3ad)https://github.com/THUDM/slime/blob/681b3adca54105d5ecd3fb822fa0dc58a427e0f9/slime/backends/megatron_utils/actor.py}

Full NCCL

For non-colocated --update-weight-mode full --update-weight-transport nccl, NCCL (the NVIDIA Collective Communications Library) carries the update. Training rank 0 pauses generation and flushes SGLang caches. Pipeline-parallel (PP) source ranks gather tensor-parallel (TP) and expert-parallel (EP) shards into Hugging Face (HF)-shaped chunks, then broadcast them to rollout engines. Engines resume after the chunks and any quantization post-processing finish. The rollout-engine lock serializes broadcast operations; it isn't a durable, cross-system transaction log.

Full NCCL avoids writing a checkpoint for each transfer, but its latency depends on gathering, network topology, and synchronization. A restarted engine needs another transfer or a separate checkpoint; the broadcast itself leaves no replay artifact.

Full disk

Full disk uses update-weight-mode=full and update-weight-transport=disk. Each sync writes a canonical Hugging Face checkpoint directory under a version such as weight_v000003. The rollout engine pulls that directory, optionally into a host-local checkpoint, and reloads through its ordinary update_weights_from_disk endpoint. A post-write hook can publish files to an object-store-backed mount before hosts read them.

A versioned directory makes the update inspectable, but not automatically durable. Full-disk reload normally removes the published directory unless --update-weight-disk-keep-files is enabled. Retention and durable storage still need explicit configuration. The file is a weight export, not a complete optimizer-and-data-source recovery checkpoint.

Publication and an optional host-local pull occur before generation is paused. Pause, cache flush, and reload form the live-weight transition. Verify that every engine reads the published bytes, then check readiness across the fleet after reload.

Delta disk

Delta mode is disk-only and non-colocated. The trainer first captures a CPU baseline seeded from --hf-checkpoint, then diffs each gathered HF tensor on later syncs. Changed bytes are compressed with zstd and written as a self-describing version. Each rollout host applies the delta into its local full checkpoint through /pull_weights, verifies per-tensor checksums, and reloads through the same disk endpoint used by full sync.

Exclusive-or (XOR) encodes byte differences; compression savings depend on how much changes, not on the mode name. A raw XOR patch must be applied once to the intended base. overwrite stores changed positions and values, so repeating the same writes doesn't toggle them back, but it still needs the correct base and version ordering. The index records those identities and a new-state checksum. Neither an xxHash checksum nor a version number authenticates an untrusted publisher.^{[9]Reference 9Slime Disk-Delta Weight Update (681b3ad)https://github.com/THUDM/slime/blob/681b3adca54105d5ecd3fb822fa0dc58a427e0f9/slime/backends/megatron_utils/update_weight/update_weight_from_disk_delta.py}

def xor_bytes(left: bytes, right: bytes) -> bytes:
    if len(left) != len(right):
        raise ValueError("Patch and base lengths must match")
    return bytes(x ^ y for x, y in zip(left, right))

base = bytes([1, 2, 3, 4])
new = bytes([1, 2, 8, 4])
delta = xor_bytes(new, base)
applied_once = xor_bytes(base, delta)
applied_twice = xor_bytes(applied_once, delta)
assert applied_once == new
assert applied_twice == base
try:
    xor_bytes(base, delta[:-1])
except ValueError:
    pass
else:
    raise AssertionError("Truncated delta accepted")
print(list(delta), list(applied_once), list(applied_twice))

[0, 0, 11, 0] [1, 2, 8, 4] [1, 2, 3, 4]

Byte 8 ^ 3 is 11. Applying the patch once reaches the new state. Applying it twice restores the base, which is why a repeated XOR is dangerous: it can look successful while serving the old model.

Keep source version, declared base, destination, checksum, reload result, and served version together. Before retrying, determine whether the local base was already patched; if its state is ambiguous, stop new admissions and repair from a known checkpoint. Don't blindly XOR again. Crash-safe recovery needs atomic persistence of bytes and version state. The four-byte example illustrates comparison only.

These are argument fragments, not a runnable shell command. Add them to a compatible non-colocated training launch with model, dataset, actor, and SGLang configuration. The shared directory must be visible to all participating hosts; the local checkpoint path is host-local:

--update-weight-mode delta \
--update-weight-transport disk \
--update-weight-disk-dir /shared/fs/delta-updates \
--update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt \
--update-weight-delta-encoding xor \
--update-weight-delta-checksum xxh3-128

View larger

Agentic customization keeps token provenance

Start agentic work with --custom-generate-function-path when one prompt should run a custom loop. The function can call tools, retrieval, a browser, a sandbox, or another service, then return one Sample or a list of sibling Samples. Use --custom-rm-path for verifier or environment reward.^{[10]Reference 10slime Customization Guidehttps://thudm.github.io/slime/get_started/customization.html[11]Reference 11Agentic Reinforcement Learning with slimehttps://thudm.github.io/slime/get_started/agent.html}

Replace the whole rollout function only when scheduling or buffering can't fit the default per-sample loop. Keeping the default orchestration leaves grouping, validation, and handoff behavior visible while the custom hook owns the environment-specific work.

Calling a tool is the smaller step. Preserving token provenance around that call is what keeps the training target honest. Model-generated output tokens should have loss_mask=1. Tool observations, template fragments, and environment text inserted into the response suffix should normally have loss_mask=0; the initial prompt sits outside that mask. The adapter records sampled token IDs and log probabilities directly from SGLang instead of decoding text and tokenizing it again.

When a reward changes, inspect this boundary before changing the objective. Compare the sampled IDs, mask, response length, verifier input, and policy version. A readable transcript can hide a different training target.

slime ships protocol adapters for that job. AnthropicAdapter and OpenAIAdapter render a chat template, call SGLang with input_ids and return_logprob=True, and export the returned IDs as trainable segments. They also pass a stable session_id as X-SMG-Routing-Key. The adapters are a convenience layer, not a second agent framework.

One agent execution can fan out into subagent or compaction segments. Return them with the same rollout_id. Choose reward normalization from the reward-to-advantage processing and loss reduction; the segment count alone does not determine it. In the pinned per-rollout-mean path, sibling losses already share the rollout's total trainable-token denominator. Scaling their advantages down again would reduce that rollout's contribution.^{[10]Reference 10slime Customization Guidehttps://thudm.github.io/slime/get_started/customization.html[12]Reference 12Slime Context-Parallel Loss Reduction (681b3ad)https://github.com/THUDM/slime/blob/681b3adca54105d5ecd3fb822fa0dc58a427e0f9/slime/backends/megatron_utils/cp_utils.py}

Hold per-token loss terms fixed at [1, 3, 5, 7]. Splitting them into two segments must not change their mean from 4 to 8, or halve it to 2. This isolates the reduction arithmetic; it doesn't simulate GRPO reward normalization or gradients:

segments = [[1.0, 3.0], [5.0, 7.0]]
denominator = sum(len(segment) for segment in segments)
one_rollout = sum(sum(segment) / denominator for segment in segments)
wrong_independent = sum(sum(segment) / len(segment) for segment in segments)
wrong_extra_halving = sum(sum(segment) / 2 / denominator for segment in segments)
assert one_rollout == 4.0
assert wrong_independent == 8.0
assert wrong_extra_halving == 2.0
# A different split preserves the same aggregate denominator.
assert sum(sum(s) / denominator for s in [[1.0], [3.0, 5.0, 7.0]]) == one_rollout
print(f"one rollout={one_rollout}; independent sum={wrong_independent}; "
      f"extra halving={wrong_extra_halving}")

one rollout=4.0; independent sum=8.0; extra halving=2.0

If segments land in different microbatches, compute the denominator over the whole training step first. A microbatch-local denominator would count one rollout repeatedly. Per-token loss modes or custom objectives have different normalization rules, so check the configured reducer before deciding how to assign segment rewards.

The coding-agent example makes this concrete. An agent edits a fresh sandbox, captures a diff, and a clean grading sandbox runs tests on that diff. The training target is still the model's token stream, not final text copied from a log. This split lets teams change the runner without changing Megatron's loss code.

The integration sketch below copies a trajectory's exact sequence, aligned mask and log probabilities, serving weight versions, and verifier reward into one Sample. run_agent, verify, and the trajectory type are application-owned and omitted; this isn't a drop-in script or a locally executed agent run.

from slime.utils.types import Sample

async def custom_generate(args, sample, sampling_params):
    trajectory = await run_agent(sample.prompt, sampling_params=sampling_params)
    sample.tokens = trajectory.prompt_ids + trajectory.response_tokens
    sample.loss_mask = trajectory.response_loss_mask
    sample.response_length = len(trajectory.response_tokens)
    sample.rollout_log_probs = trajectory.response_log_probs
    sample.weight_versions = list(trajectory.weight_versions)
    sample.reward = await verify(trajectory)
    if sample.rollout_id is None:
        sample.rollout_id = sample.index
    if sample.rollout_id is None:
        raise ValueError("A trajectory ID is required")
    sample.status = Sample.Status.COMPLETED
    return sample

Tool permissions are application-owned. Preserve the weight-version metadata from the calls that generated each trajectory; a final engine health check can't reconstruct it. Use the real status enum, and test is None when choosing an identity because rollout_id=0 is valid. Mark completion only after the chosen terminal condition and verifier succeed; failures need an explicit retry/drop policy.

Applications and ecosystem

The pinned README identifies slime as infrastructure for the GLM family and includes launch examples for other model families. These examples document project-reported adoption. For memory, throughput, or quality comparisons, read the model-specific launch script and report, then reproduce the workload on the target configuration.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime}

Choose a starting example by the boundary your workload stresses: verifier quality, multi-turn context, tool latency, or weight transport. Evaluate coding-sandbox permissions and aborted-generation handling separately from a math-verifier run.

Treat each application as an experiment with a receipt. Save resolved launch arguments, model checkpoint, data revision, reward code, sync mode, topology, policy versions, rollout dumps, and metrics. Without that receipt, a green reward curve can't tell you whether the policy improved or the verifier changed.

Strengths, limits, and failure paths

slime's strengths come from narrow ownership boundaries. It doesn't hide Megatron parallelism behind a new trainer abstraction or flatten SGLang's serving flags to a lowest common denominator. Upstream engine work can land without a second trainer API, while RL-specific sample and synchronization rules stay in one place.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

Most production failures cross a boundary. A rollout server can stay alive while serving old weights. A reward function can return a number while masking every response token. A fully-async queue can stay busy while repeatedly requeueing aborted groups. Name the boundary first, then inspect the receipt that crosses it.

Fault tolerance and reproducibility

In the pinned snapshot, --use-fault-tolerance starts heartbeat checks against SGLang servers. If a heartbeat times out, slime stops the unhealthy engine. After the current rollout round finishes, it restarts the engine and applies the correct parameters before future requests. This covers engine recovery between rollout rounds. A failed trainer rank or preempted cluster job needs checkpoint recovery.^{[5]Reference 5slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

The current recovery guide, checked October 10, 2026, describes a broader boundary. Health recovery is enabled by default for serving clusters started by slime; --use-fault-tolerance remains a compatibility flag. A named, detached serving actor can survive a failed training attempt while the Ray cluster remains alive. With straw storage or versioned rollout debug dumps, resubmitting training can replay retained batches after the last committed model checkpoint. Ordinary stateful optimization requires optimizer and RNG state as well as model state. Without retained rollout data, keeping serving alive does not reconstruct those batches.^{[5]Reference 5slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

For example, suppose checkpoint 10 is committed, batches 11 and 12 train, and the trainer fails before checkpoint 12 commits. Recovery must restore checkpoint 10 and retrain the retained batches, including batch 12 even if its earlier optimizer step finished. Skipping directly to batch 13 would combine old checkpoint weights with advanced data progress. Keep the same serving session identity and compatible rollout configuration, resubmit explicitly, and inspect replay receipts. Losing the serving actor or Ray cluster still requires fresh serving and checkpoint recovery. Changing parallel layouts can also change updates even when endpoints persist.^{[5]Reference 5slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

Debug replay narrows the search. Save a rollout with --save-debug-rollout-data, load it with --load-debug-rollout-data, and use --debug-train-only to replay conversion and training without starting SGLang. Pair the dump with checkpoints, trace spans, and a pinned launch command. Load only trusted dumps: the pinned replay path uses PyTorch deserialization with weights_only=False, not a safe parser for arbitrary uploads.

Recovery has its own false positives. Health checks can mistake first-run kernel compilation for a dead server, so the docs expose a first-wait setting. Fully-async workers can hide slow samples in a warm queue, so log queue length and completion age. Disk sync can hide stale mounts, so verify every host's local checkpoint and weight version.

The repository's continuous integration (CI) mirrors this split. CPU tests cover Sample behavior, rollout validation, argument contracts, and customization hooks. GPU end-to-end tests cover Megatron, SGLang deployment, async rollout, checkpointing, precision, and replay. Include an environment-specific multi-node smoke run that verifies sync visibility in the release receipt.

Project identity

slime is published through the THUDM organization, whose official profile identifies its THUKEG and Z.ai lineage.^{[13]Reference 13THUDMhttps://github.com/thudm} The repository citation names Zilin Zhu, Chengxing Xie, Xin Lv, and slime contributors. The introductory blog frames the project around an SGLang-native rollout path, Megatron training, Ray resource management, and custom data generation.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

The first release notes describe v0.1.0 as focusing on MoE inference, memory offload, faster parameter updates, Megatron parallel strategies, and strict correctness checks. Those notes describe that release; check the current branch before relying on the same performance or feature set.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime}

Use the pinned source for implementation behavior and the LMSYS design post for architectural motivation. Model reports and related systems papers answer different questions. Validate each framework path with its own source inspection and tests.

Contribution policy is intentionally focused. Bug fixes and general RL optimizations that can be verified through CI are welcome. Large refactors, universal agent abstractions, and changes that can't be tested against routine training stay outside core scope. That governance keeps internal and open development aligned while leaving application-specific systems in their own repositories.^{[14]Reference 14Contributing to slimehttps://github.com/THUDM/slime/blob/aaf5c2092b01219fa0d5c2d323741d409086ca32/CONTRIBUTING.md}

That history explains the project's shape. slime doesn't own every environment, reward model, or serving backend. It concentrates on the handoff between training and rollout, then lets teams compose the rest around stable contracts. Which file owns each side of that handoff?

A source and code reading route

Read one vertical path before opening every module. This walkthrough uses the official slime repository at commit 681b3adca54105d5ecd3fb822fa0dc58a427e0f9.^{[3]Reference 3slime Source Snapshot (681b3ad)https://github.com/THUDM/slime/tree/681b3adca54105d5ecd3fb822fa0dc58a427e0f9} Start with the README architecture section and introductory blog to see why Ray, Megatron, SGLang, and the Data Buffer are separate. Then open train.py and train_async.py side by side. Mark each ray.get, generate.remote, and update_weights call. That is the timing model in executable form.

Next inspect slime/ray/placement_group.py and slime/ray/rollout.py. Follow how bundles become server groups, how routers start, and how RolloutManager.generate() converts nested Samples into train data. Read slime/rollout/sglang_rollout.py for prompt IDs, session routing, aborts, and reward calls. Compare slime/rollout/fully_async_rollout.py with the fully-async example to see queue lifetime and abort requeue.

For the weight boundary, read slime/ray/actor_group.py, then the transport modules under slime/backends/megatron_utils/update_weight/: update_weight_from_distributed.py, update_weight_from_disk.py, and update_weight_from_disk_delta.py. Start with actor-group lifecycle. The transport modules show gathering, publication, and reload orchestration. For engine-side delta application, follow pull_weights in slime/backends/sglang_utils/sglang_engine.py through its management endpoint and helpers. Keep a paper notebook with source version, base version, and engine state for one hypothetical sync.

For agents, read the published customization and agent guides alongside examples/coding_agent_rl. Compare their examples with the pinned Sample type and get_sum_of_sample_mean reducer before adapting a reward rule. For operations, use the fault-tolerance guide to choose a recovery experiment, then inspect the corresponding implementation and verify restored state alongside service health.^{[10]Reference 10slime Customization Guidehttps://thudm.github.io/slime/get_started/customization.html[11]Reference 11Agentic Reinforcement Learning with slimehttps://thudm.github.io/slime/get_started/agent.html[5]Reference 5slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

Carry one question into every repository: which state is authoritative at this step? In slime, authority sits in token IDs during rollout, grouped Samples in the buffer, Megatron parameters during training, and a versioned sync artifact before SGLang serves again.

Before Megatron steps, line up rollout 7's reward of 1.0 with the mask, the rollout ID, and the served weight version. If they agree, Megatron can take the step.

What to remember

• Ray placement groups make train and rollout topology explicit, including colocation, decoupling, and external engines.
• SGLang and sgl-router generate token evidence; Megatron consumes that evidence for distributed actor training.
• The Data Buffer contract is token based. Preserve log probabilities, loss masks, rewards, statuses, and rollout IDs.
• Synchronous training waits for each round. Async training overlaps one future rollout. Fully async keeps a warm worker and queue across boundaries.
• Colocated full sync uses CUDA IPC. Non-colocated NCCL streams chunks; disk paths publish full weights or checked deltas before live reload.
• Agent hooks can add tools, sandboxes, search, and branches without changing the training kernel, but token provenance remains the correctness rule.
• The pinned health monitor restarts rollout engines. Newer retained-serving recovery can replay saved batches after a trainer failure, while cluster loss still needs checkpoint and scheduler recovery.
• Read code by following one Sample and one weight version from source to engine, then verify claims against first-party docs.

Evaluation rubric

• Trace one trajectory's token IDs, response mask, log probabilities, status, and serving-policy versions through rollout and training.
• Distinguish independent attempts from segments of one execution, and calculate the correct loss denominator across microbatches.
• Explain what ordinary async waits for and what a persistent custom rollout worker must coordinate separately.
• Choose a weight-sync path for a stated topology and identify its base-version, visibility, retention, and recovery checks.

Follow-up questions