Deep Dive - slime

A model can produce a promising answer and still learn from the wrong token sequence. In reinforcement-learning (RL) post-training, generation, reward computation, training, and weight updates form one moving system. A boundary bug can resemble a bad algorithm while its true cause is stale weights, a missing loss mask, or a rollout counted twice.

slime is an open framework for this loop. It places Ray around Megatron training and SGLang rollout servers, then connects them through a Data Buffer path that preserves prompts, sampled tokens, log probabilities, rewards, and grouping metadata. The project keeps those boundaries visible so a team can change its environment or agent without replacing the training kernel.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

This lesson follows one rollout from resource placement to a train step, then compares three timing models and three weight-sync paths. It also shows where agentic code enters, which failures the repository treats as first-class, and how to read the source without getting lost in launch scripts.

Why slime exists

RL post-training is an online loop, not a static dataset pass. A prompt is sent to a policy, the policy samples one or more responses, an environment or verifier assigns reward, and a trainer changes the policy. The next prompt should see the new policy, but the system also needs enough overlap to keep GPUs busy. Long responses, tool calls, and failed environments make each iteration irregular.

Many teams respond by creating separate stacks for math, tool use, multi-turn agents, asynchronous sampling, and supervised fine-tuning. Those forks share a training bug only after someone copies a patch into every branch. slime takes a different boundary: keep the training and serving engines close to their upstream interfaces, and let users inject data-generation and reward logic at explicit hooks.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

The design has two deliberate opinions. Megatron owns high-throughput parameter updates. SGLang owns token generation behind a router. Ray owns placement and actor lifecycle. slime owns the handoff, the sample contract, the timing policy, and the checks that keep rollout data aligned with training.

The result isn't a universal agent framework. An agent can use a sandbox, search service, browser, tool server, or another model, but slime only needs its generated tokens and reward to arrive in the expected shape. That narrow contract is why the core can stay small while applications differ.

One mental model: a token ledger between two engines

A rollout acts as a ledger entry. Its prompt and sampled response carry the sequence, while log probabilities, loss masks, reward, status, and rollout ID explain how that sequence should affect the next gradient update. Ray decides where the ledger work runs, while the Data Buffer keeps entries available until the trainer consumes a complete group.

The policy appears twice, but with different jobs. SGLang reads current weights and produces tokens quickly. Megatron reads those tokens and computes losses, gradients, and optimizer updates. A weight-sync operation is the commit that changes what SGLang will sample next. If that commit is delayed, data becomes off-policy by design, so the timing policy must be explicit.

The loop's arrow back to SGLang is more important than the boxes. A generated response isn't training data until its token IDs, response length, loss mask, reward, and status agree. A weight update isn't complete until the rollout engines have loaded the intended version and can serve again. The same diagram applies to a math verifier and a coding agent, even though their custom logic differs.

Ray placement makes topology a contract

Ray placement groups reserve bundles such as one GPU plus one central processing unit (CPU) slot. slime builds one placement group and records which bundles belong to actor training and which belong to rollout. Sorting bundles by node and physical GPU keeps the logical rank order stable when Ray schedules a distributed cluster. The placement code is infrastructure, not model logic, but it decides whether a chosen sync path is legal.

With --colocate, actor training and rollout share a GPU allocation. The loop must offload one side before the other uses the memory. Without colocation, actor and rollout receive separate bundles and can overlap more freely, at the cost of extra hardware and a transport boundary for weights. Async training requires the decoupled shape in train_async.py; its first assertion rejects colocation.

The same placement group can also describe server groups with different sizes. SGLang configuration supports regular, prefill, decode, and encoder groups, while Ray still accounts for every GPU slot. A heterogeneous layout changes where requests run, not what a Sample means.

slime passes Megatron arguments directly and prefixes installed SGLang arguments with --sglang-. That keeps tensor, pipeline, expert, context, and data parallel settings visible to Megatron, while serving flags stay available to SGLang. A launch file can therefore be specific about topology without adding a second configuration language for every upstream option.^{[3]Reference 3slime Documentationhttps://thudm.github.io/slime/}

SGLang and sgl-router own rollout

slime launches SGLang in server mode and keeps a router endpoint in front of the workers. A regular sample is tokenized, sent to /generate with return_logprob=True, and appended with response token IDs, per-token log probabilities, and metadata. The decoded text is useful for inspection, but the token IDs are the training source of truth.

The router also gives multi-turn sessions a stable route. When router_policy is consistent_hashing, slime sends X-SMG-Routing-Key from the sample session ID. One session can therefore return to the same worker and make better use of its prefix cache. This helps latency, but it doesn't make a session correct by itself. The trajectory manager still has to prove which tokens came from the model.

The default rollout function creates groups of samples, runs generation concurrently, computes rewards, and waits until the requested batch is valid. Dynamic sampling can discard groups whose reward variation is unhelpful. Partial rollout mode can abort requests and keep unfinished samples for a later pass. SGLang's /abort_request endpoint exists for this kind of early stop when oversampling has already found enough valid responses.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

There are two useful debug boundaries. --debug-rollout-only exercises SGLang and saves generated data without training. --debug-train-only loads the training side without starting rollout servers. If a reward or tokenization change breaks a run, those modes tell you whether the fault begins before or after the Data Buffer.

Data Buffer is the handoff contract

The logical Data Buffer is the bridge from prompt initialization to generated samples. A data source supplies groups, the rollout function fills them, and the manager converts the resulting Samples into tensors and schedules them across data-parallel ranks. The optional slime_plugins/rollout_buffer runs a separate HTTP service for agent trajectory generation, but it still returns the same kind of grouped evidence.

Each Sample carries more than text. tokens contains prompt and response IDs, response_length tells the trainer where generated tokens end, and loss_mask marks which response positions should contribute to loss. reward can come from a built-in math scorer or a custom verifier. status distinguishes completed, truncated, and aborted work. rollout_id ties sibling segments to one rollout for loss aggregation.

The manager normalizes rewards by group for estimators such as Group Relative Policy Optimization (GRPO) when configured, converts fields to CPU tensors, and computes a data-parallel schedule. It can split a compact agent rollout into multiple Samples, but all siblings must share one rollout_id. That invariant prevents a long trajectory from gaining extra weight merely because context compaction created more segments.

An external rollout buffer adds another boundary. Its generators write items over HTTP, group them by instance ID, and expose valid groups to the trainer. If the service loses a process or clears temporary data, the trainer may still be healthy while its next batch is empty. Treat buffer availability, group completeness, and metadata freshness as separate health signals.

Megatron owns the train step

Megatron receives the scheduled token tensors from Ray's object store or tensor transport. Its actor workers initialize distributed process groups, create model and optimizer state, compute log probabilities, and run the chosen RL loss. A critic model can run alongside the actor, but GRPO-style paths can train without one when relative rewards provide the advantage signal.

The actor group exposes async_train, save_model, and update_weights. The name async_train refers to the Ray call returning references, not to a fully asynchronous RL policy. train.py still waits for those references before saving or updating rollout weights. Read the caller before inferring timing from a method name.

Megatron parallelism remains native. Tensor, pipeline, expert, context, and data parallel groups are created by Megatron, while slime converts gathered parameters to the format SGLang expects. The bridge can use raw conversion or a Megatron Bridge path. This reuse keeps optimizer and checkpoint behavior close to Megatron, but it also means version drift in either upstream engine can surface at the boundary.

The smallest useful inspection is a batch receipt:

Expected output is 3. The first token is prompt context, while the three response positions are eligible for loss. A real Sample also stores response text, reward, and log probabilities, but this tiny assertion catches a common adapter bug before a GPU run.

Three timing models

Timing controls how much new data is collected before a policy update and how much rollout work may overlap training. The names are easy to confuse, so tie each one to a concrete loop in the repository.

Synchronous: finish, train, sync

train.py asks the RolloutManager for one rollout, waits for its data, trains on it, and then calls actor_model.update_weights(). If rollout and training are colocated, it can offload memory between those phases. The next generation sees the updated actor after the sync completes. This path is easiest to reason about because one loop iteration has one visible policy version.

The cost is idle time. A slow verifier or one long response holds the batch open, and training can't use GPUs while generation is still collecting that batch. Dynamic sampling and partial aborts reduce some waste, but the loop remains round-bound.

Async: start next rollout early

train_async.py starts rollout_manager.generate.remote(next_id) before training the current rollout. Training and generation occupy separate GPU ranges, and a later update_weights_interval controls how often the actor waits for the in-flight generation before changing weights. The code explicitly drains that future before a sync so a server never sees parameters change mid-generation.

Async reduces idle time, but samples can be generated by an older policy. That staleness is a chosen property, not a hidden bug. Compare the rollout's weight version with the actor version in logs, and don't increase overlap until reward and Kullback-Leibler (KL) behavior stay within your experiment's contract.

Fully async: keep a warm worker

The fully-async example uses train_async.py plus slime.rollout.fully_async_rollout.generate_rollout_fully_async. A process-wide thread owns an asyncio loop, keeps a fixed number of generate_and_rm_group tasks in flight, and draws new groups from the global data buffer as soon as slots open. Completed groups sit in an output queue until the next training call needs its target batch.

This worker decouples in-flight concurrency from one rollout's batch size. It sorts completed groups by sample index for deterministic handoff, requeues any group containing an aborted Sample, and doesn't support evaluation mode. A long-tail agent can continue while a later training step consumes already completed work, but policy staleness and queue backpressure become operational metrics.

Weight synchronization is a correctness boundary

Training and rollout can use separate processes, hosts, or even GPU types. The actor therefore needs a transport that moves an exact parameter state to SGLang and a lifecycle that prevents requests from reading half an update. slime has three meaningful paths: full NCCL, full disk, and delta disk.

Full NCCL

This is the default update-weight-mode=full with update-weight-transport=nccl. NCCL, the NVIDIA Collective Communications Library, carries the update. Training rank 0 pauses generation and flushes SGLang caches. Pipeline-parallel (PP) source ranks gather tensor-parallel (TP) and expert-parallel (EP) shards into Hugging Face (HF)-shaped chunks and broadcast those chunks to rollout engines. Engines resume only after the final chunk and any quantization post-processing complete. An actor-group lock prevents a competing update from opening another broadcast sequence.

Full NCCL has low coordination overhead when training and rollout share a compatible network and process setup. It isn't a disk artifact, so a restarted external engine can't replay the update without another transfer. Colocation also makes this path sensitive to memory pressure and CUDA interprocess communication (IPC) constraints.

Full disk

Full disk uses update-weight-mode=full and update-weight-transport=disk. Each sync writes a canonical Hugging Face checkpoint directory under a version such as weight_v000003. The rollout engine pulls the directory, optionally into a host-local checkpoint, and reloads through its ordinary update_weights_from_disk endpoint. A post-write hook can publish files to an object-store-backed mount before hosts read them.

Disk makes the version a durable artifact. It supports external rollout engines and release-train flows, but it adds filesystem visibility, storage cleanup, and checkpoint write time. A successful write on one host isn't proof that every engine can read the same bytes.

Delta disk

Delta mode is disk-only and non-colocated. The trainer first captures a CPU baseline seeded from --hf-checkpoint, then diffs each gathered HF tensor on later syncs. Changed bytes are compressed with zstd and written as a self-describing version. Each rollout host applies the delta into its local full checkpoint, verifies per-tensor checksums, and reloads through the same disk endpoint used by full sync.

The exclusive-or (XOR) encoding is small and fast, but it must be applied exactly once to the declared base. overwrite stores changed positions and new values, so re-applying a version converges instead of toggling back. The index records base version, encoding, compression, and checksum algorithm. A wrong base or out-of-order version fails loudly instead of serving a plausible but corrupt model.^{[3]Reference 3slime Documentationhttps://thudm.github.io/slime/}

Agentic customization keeps token provenance

Start agentic work with --custom-generate-function-path when one prompt should run a custom loop. The function can call tools, retrieval, a browser, a sandbox, or another service, then return one Sample or a list of sibling Samples. Use --custom-rm-path for verifier or environment reward. Replace the whole rollout function only when scheduling or buffering can't fit the default per-sample loop.^{[4]Reference 4slime Customization Guidehttps://thudm.github.io/slime/get_started/customization.html[5]Reference 5Agentic Reinforcement Learning with slimehttps://thudm.github.io/slime/get_started/agent.html}

Calling a tool is straightforward; preserving token provenance around that call takes care. Model-generated output tokens should have loss_mask=1; prompt templates, tool observations, and environment text should normally have loss_mask=0. The adapter records sampled token IDs and log probabilities directly from SGLang instead of decoding text and tokenizing it again.

One agent execution can fan out. A subagent branch, context compaction boundary, or main-agent continuation may become several trainable segments. Return them with the same rollout_id. If one trajectory has one total reward, custom code or a reward postprocessor must assign reward / K across its K segments so branch count doesn't amplify the trajectory. The manager validates IDs, but it doesn't divide reward automatically.

The coding-agent example makes this concrete: a harness edits a fresh sandbox, captures a diff, and grades that diff in a second clean sandbox. The training target is still the model's token stream, not the final text copied from a log. This split lets teams change the harness without changing Megatron's loss code.

This sketch omits adapter setup and status enums, so it isn't a drop-in script. It does show the contract: tokens contains prompt plus generated IDs, response-side masks and log probabilities follow sampled token provenance, reward comes from an explicit verifier, and the rollout ID remains stable.

Applications and ecosystem

The repository lists slime behind the GLM-4.5 and GLM-5 model lines, along with support for Qwen, DeepSeek, and Llama families. Those references establish where the framework is used, not a universal performance ranking. Read model-specific launch scripts and reports before comparing throughput or quality.^{[6]Reference 6GLM-4.5: Agentic, Reasoning, and Coding Foundation Modelshttps://arxiv.org/abs/2508.06471[7]Reference 7GLM-5: From Vibe Coding to Agentic Engineeringhttps://arxiv.org/abs/2602.15763}

Applications span verifiable math, search, tool use, coding agents, multi-agent systems, on-policy distillation, and multimodal environments. Community projects extend the same substrate in different directions: vime swaps in a vLLM rollout backend, Relax separates actor and rollout services, and APRIL studies active partial rollouts for long-tail generation. The common point is the data and weight contract, not a shared application API.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime[8]Reference 8APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-Tail Generationhttps://arxiv.org/abs/2509.18521}

Treat each application as an experiment with a receipt. Save the launch arguments, model checkpoint, data revision, reward code, sync mode, rollout dumps, and metrics. Without that receipt, a green reward curve can't tell you whether the policy improved or the verifier changed.

Strengths, limits, and failure paths

slime's strengths come from its narrow ownership boundaries. It doesn't hide Megatron parallelism behind a new trainer abstraction, and it doesn't flatten SGLang's serving flags to a lowest common denominator. Users get a framework that can follow upstream engine work while keeping one place for RL-specific sample and synchronization rules.^{[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

Most production failures cross a boundary. Rollout servers can stay alive while serving old weights. Reward functions can return a number while masking every response token. Fully-async queues may stay busy while repeatedly requeueing aborted groups. Debugging starts by naming the boundary, then checking the receipt that crosses it.

Fault tolerance and reproducibility

--use-fault-tolerance starts heartbeat checks against SGLang servers. A timed-out engine is stopped, and after the current rollout round finishes, slime restarts it and applies the correct parameters before future requests. This is rollout-engine recovery, not a promise that a failed trainer rank or a preempted cluster job can resume from memory.^{[9]Reference 9slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html}

Debug replay narrows the search. Save a rollout with --save-debug-rollout-data, load it with --load-debug-rollout-data, and use --debug-train-only to replay conversion and training without starting SGLang. Pair replay dumps with checkpoints, trace spans, and a pinned launch command. A reproducible failure is more useful than a dashboard screenshot.

Long-running jobs should also watch backpressure. Health checks can mistake first-run kernel compilation for a dead server, so the docs expose a first-wait setting. Fully-async workers can hide slow samples in a warm queue, so log queue length and completion age. Disk sync can hide stale mounts, so verify every host's local checkpoint and weight version.

The repository's continuous integration (CI) mirrors this split. CPU tests cover Sample behavior, rollout validation, argument contracts, and customization hooks. GPU end-to-end tests cover Megatron, SGLang deployment, async rollout, checkpointing, precision, and replay. A passing unit test doesn't prove a multi-node sync is visible, so keep a small environment-specific smoke run in the release receipt.

Project identity

slime is published through the THUDM organization, whose official profile identifies its THUKEG and Z.ai lineage.^{[10]Reference 10THUDMhttps://github.com/thudm} The repository's citation names Zilin Zhu, Chengxing Xie, Xin Lv, and slime contributors. The introductory blog frames the project around an SGLang-native rollout path, Megatron training, Ray resource management, and custom data generation.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime[2]Reference 2slime: An SGLang-Native Post-Training Framework for RL Scalinghttps://www.lmsys.org/blog/2025-07-09-slime/}

The first release notes describe v0.1.0 as focusing on MoE inference, memory offload, faster parameter updates, Megatron parallel strategies, and strict correctness checks. Treat those statements as release context, not as a promise that every current branch has the same performance or feature set.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime}

The source snapshot has no dedicated peer-reviewed slime system paper. Cite the repository and LMSYS design post for its architecture, then use model and systems papers such as GLM-4.5, GLM-5, and APRIL as evidence for applications and related mechanisms. Don't turn those papers into proof of every framework claim.

Contribution policy is intentionally focused. Bug fixes and general RL optimizations that can be verified through CI are welcome. Large refactors, universal agent abstractions, and changes that can't be tested against routine training stay outside the core scope. That governance keeps internal and open development aligned while leaving application-specific systems in their own repositories.^{[11]Reference 11Contributing to slimehttps://github.com/THUDM/slime/blob/aaf5c2092b01219fa0d5c2d323741d409086ca32/CONTRIBUTING.md}

This history explains the project's shape. slime is not trying to own every environment, reward model, or serving backend. It concentrates on the hard handoff between high-throughput training and high-throughput rollout, then lets teams compose the rest around stable contracts.

A source and code reading route

This walkthrough uses the official slime repository at commit aaf5c2092b01219fa0d5c2d323741d409086ca32.^{[1]Reference 1slime: An LLM Post-Training Framework for RL Scalinghttps://github.com/THUDM/slime} Read one vertical path before opening every module. Start with the README architecture section and the introductory blog to learn why Ray, Megatron, SGLang, and the Data Buffer are separate. Then open train.py and train_async.py side by side. Mark each ray.get, each generate.remote, and each update_weights call; that is the timing model in executable form.

Next inspect slime/ray/placement_group.py and slime/ray/rollout.py. Follow how bundles become server groups, how routers are started, and how RolloutManager.generate() converts nested Samples into train data. Read slime/rollout/sglang_rollout.py for prompt IDs, session routing, aborts, and reward calls. Then compare slime/rollout/fully_async_rollout.py with the fully-async example to see queue lifetime and abort requeue.

For the weight boundary, read slime/ray/actor_group.py, update_weight_from_distributed.py, update_weight_from_disk.py, and update_weight_from_disk_delta.py in that order. The first shows the caller's lifecycle, the next two show transport, and the last shows baseline, delta, checksum, and apply ordering. Keep a paper notebook with version, base version, and engine state for one hypothetical sync.

For agents, follow docs/en/get_started/customization.md, docs/en/get_started/agent.md, and examples/coding_agent_rl. For operations, read docs/en/advanced/fault-tolerance.md, the debug guide, and the CI guide. For research context, use the slime2025blog design post, slime2026docs architecture and sync pages, slime2026customization, slime2026agents, and slime2026faulttolerance; then read the primary GLM-4.5, GLM-5, and APRIL sources before repeating any benchmark or model claim.^{[3]Reference 3slime Documentationhttps://thudm.github.io/slime/[4]Reference 4slime Customization Guidehttps://thudm.github.io/slime/get_started/customization.html[5]Reference 5Agentic Reinforcement Learning with slimehttps://thudm.github.io/slime/get_started/agent.html[9]Reference 9slime Rollout Fault Tolerancehttps://thudm.github.io/slime/advanced/fault-tolerance.html[6]Reference 6GLM-4.5: Agentic, Reasoning, and Coding Foundation Modelshttps://arxiv.org/abs/2508.06471[7]Reference 7GLM-5: From Vibe Coding to Agentic Engineeringhttps://arxiv.org/abs/2602.15763[8]Reference 8APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-Tail Generationhttps://arxiv.org/abs/2509.18521}

Carry one question into every repository: which state is authoritative at each step? In slime, authority sits in token IDs during rollout, grouped Samples in the buffer, Megatron parameters during training, and a versioned sync artifact before SGLang serves again.

What to remember

• Ray placement groups make train and rollout topology explicit, including colocation, decoupling, and external engines.
• SGLang and sgl-router generate token evidence; Megatron consumes that evidence for distributed actor training.
• The Data Buffer contract is token based. Preserve log probabilities, loss masks, rewards, statuses, and rollout IDs.
• Synchronous training waits for each round. Async training overlaps one future rollout. Fully async keeps a warm worker and queue across boundaries.
• Full NCCL streams chunks, full disk publishes a checkpoint, and delta disk applies checked patches to a declared base.
• Agent hooks can add tools, sandboxes, search, and branches without changing the training kernel, but token provenance remains the correctness rule.
• Fault tolerance restarts rollout engines and supports replay; it doesn't replace cluster-level checkpoint and scheduler recovery.
• Read code by following one Sample and one weight version from source to engine, then verify claims against first-party docs.