Deep Dive - SkyRL

A step-12 SQL-repair run can hit this boundary: a trajectory is still waiting on its database when training finishes an update. Two serving replicas load the new weights. A third answers the next turn with the old ones. The query can still earn a good reward, yet the run no longer knows which model version produced each action.

Follow one trajectory, query-42_0, from prompt through generator, environment, trainer, and weight sync. Its token IDs, loss masks, reward, and identity must survive every handoff. Version tracking needs extra care: a scheduled trainer step is recorded for each async group, but that number isn't a per-token record of the weights used during an in-flight update.

The implementation walkthrough is pinned to official SkyRL commit 955c3e23bf8939b28a9d6c424308a762dcb8a2e7. Its selected source contracts and living documentation were rechecked September 23, 2026. It uses the unified skyrl/ package; skyrl-gym and skyrl-agent remain separate directories. Older skyrl-train and skyrl-tx recipes use different paths. Match a recipe to its commit before copying commands.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Replaceable interfaces, not one trainer

Suppose query-42_0 becomes a terminal task tomorrow. Which code should change? The environment may change from SQL execution to shell commands, and the generator may change how it parses turns. Advantage calculation should keep receiving the same trajectory contract. That is the boundary to look for before opening a trainer file.

SkyRL names four working seams: trajectory generator, environment, inference engine, and trainer. A controller role also owns placement, initialization, and control flow, although current training control still lives in trainer.py while extraction of a standalone controller remains work in progress. Explicit inputs and outputs keep the trainer from knowing every task implementation.^{[2]Reference 2SkyRL Overviewhttps://docs.skyrl.ai/docs/getting-started/overview}

This makes an extension local. A Text-to-SQL task adds an environment or generator; it doesn't edit advantage calculation. A local vLLM worker can be replaced by a remote OpenAI-compatible endpoint without moving reward processing. One-step pipelining can become fully asynchronous while the trajectory fields stay recognizable. The schedule and freshness policy change, but the handoff still answers the same ledger questions.

The native trainer includes Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and several policy-gradient variants. Its documented fully async backends are Fully Sharded Data Parallel (FSDP) and Megatron. The Tinker path also includes JAX; backend presence doesn't imply every trainer supports it. HTTP inference uses vLLM API servers plus a router.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

💡 Key insight: If a seam preserves prompt, token, reward, mask, and version semantics, the implementation behind it can change without changing the learning loop.

Project identity and lineage

The seams are code boundaries, not an org chart. SkyRL is developed at the Berkeley Sky Computing Lab in collaboration with Anyscale. Its repository acknowledges compute support from Databricks, NVIDIA, Lambda Labs, AMD, AWS, Modal, and Daytona. Those acknowledgments explain context. They don't assign every subsystem to every organization.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

The lineage is visible in the repository layout and release notes:

The project also credits ideas and code from veRL, OpenRLHF, Search-R1, OpenReasonerZero, and NeMo-RL. That lineage maps the ecosystem. It doesn't make SkyRL a drop-in replacement for any one project, and it doesn't say what the trainer still needs after two SQL turns.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Generator, environment, trainer

Return to query-42_0. Before naming classes, predict what the trainer needs after two SQL turns: the sampled response, the database observation, the reward, and a way to distinguish trainable tokens from context. If that record is complete, the optimizer can stay unaware of whether the environment is SQL, a terminal, or a browser.

A controller role sets up execution. A Generator coordinates inference and environments to produce trajectories. A Trainer turns those trajectories into gradients. Current control flow remains in trainer.py, while InferenceEngine and Environment stay replaceable boundaries inside the Generator.

The loop isn't "model in, loss out." A multi-turn trajectory can include token IDs, tool calls, observations, rewards, per-token masks, and a stop reason. Those fields carry the decision from environment execution into optimization. Without them, a trainer can apply loss to an observation, credit a failed tool call as success, or join a retry to the wrong episode.

Vocabulary at the seams

The table is a decoder ring for the code. Read Rollout as the artifact being handed forward, and read the final column as the bug that appears when one boundary silently changes its meaning.

GeneratorInput includes prompts, environment classes, optional environment extras, sampling parameters, trajectory IDs, and batch metadata. GeneratorOutput returns batched prompt and response token IDs, rewards, loss masks, stop reasons, rollout metrics, optional log probabilities, trajectory IDs, and timing splits. It also includes step-wise completion flags and vision tensors. TypedDict describes the interface for type checkers; it doesn't validate a custom generator's arrays at runtime.^{[7]Reference 7SkyRL Generator Contract (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/base.py}

Here Optional[...] means a field's value may be None; it doesn't make that key optional. The pinned TypedDict uses required keys. A real extension must populate its declared fields, using None where appropriate, as well as preserve array alignment.^{[7]Reference 7SkyRL Generator Contract (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/base.py}

The architecture in one request

The same handoff must work for a one-turn math answer and a long tool-use task. A generator may call an environment many times before returning one output, but the trainer still receives one normalized object. That normalization is what lets us reuse the rest of the route for query-42_0.

View larger

The two feedback paths have different jobs: inference samples actions, while the environment executes them and returns observations. The trainer receives completed records rather than raw tool traffic. Zero-variance rewards, broken tools, stale groups, and failed transfers need separate handling at those boundaries.

A concrete trajectory

Now follow the ledger instead of the class diagram. The prompt is: "List active customers with more than three unresolved incidents." The generator asks for a SQL action, sends it to a sandboxed database environment, appends either an error or rows as an observation, and gives the model another turn to repair its query. The environment supplies evidence; the loss mask decides which tokens can move the policy.

Call this trajectory query-42_0. Should the database error receive policy loss? No: it's context, not a sampled action. To keep the alignment visible, use a toy vocabulary with two tokens for an attempted query, one observation token, and two tokens for its repair. These IDs don't come from a real tokenizer, and the labels aren't executable SQL. This reduced view shows one row of a batched output; other declared fields are omitted for legibility, so it isn't a complete typed return object:

The reward field accepts either one scalar per trajectory or one reward per response token. It isn't an arbitrary list of turn rewards: the default multi-turn generator stamps each turn reward onto the corresponding final action token. Here the total is 0.8; aggregation and advantage estimation decide how that return influences the sampled actions.^{[7]Reference 7SkyRL Generator Contract (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/base.py[8]Reference 8SkyRL Gym Generator (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/skyrl_gym_generator.py}

View larger

The IDs and masks aren't decoration. SkyRLGymGenerator enforces token-in-token-out in its default multi-turn path: it appends sampled token IDs, appends tokenized observations with mask 0, then samples again. If you retokenize the whole chat from strings, token boundaries can change after the sampler has finished. Training then sees a different sequence from the one that earned the reward.^{[9]Reference 9SkyRL-Gym Generatorhttps://docs.skyrl.ai/docs/tutorials/skyrl_gym_generator}

A multi-turn batch can flatten several conversations. Step-wise training can return one turn at a time and mark whether an episode is complete. Keep those distinctions attached to each row, or a retry and its reward can be joined to the wrong turn.

The toy row below checks alignment and computes the masked mean negative log probability for sampled actions. GRPO training adds group-relative advantages and clipped policy-ratio terms; this alignment check leaves those terms out. Changing the observation's supplied score doesn't change this objective, while changing a sampled action's score does.

from math import isclose, isfinite

token_ids = [11, 12, 90, 21, 22]
masks = [1, 1, 0, 1, 1]
rewards = [0, -0.2, 0, 0, 1.0]
assert len(token_ids) == len(masks) == len(rewards)

def masked_nll(logprobs, mask):
    if len(logprobs) != len(mask) or any(m not in (0, 1) for m in mask):
        raise ValueError("unaligned or nonbinary mask")
    selected = [lp for lp, m in zip(logprobs, mask) if m]
    if not selected or not all(isfinite(lp) and lp <= 0 for lp in selected):
        raise ValueError("need finite action log probabilities and a nonempty mask")
    return -sum(selected) / len(selected)

logprobs = [-0.2, -0.4, 0.0, -0.6, -0.8]
baseline = masked_nll(logprobs, masks)
changed_observation = logprobs[:]
changed_observation[2] = float("nan")
assert masked_nll(changed_observation, masks) == baseline
changed_action = logprobs[:]
changed_action[4] = -1.2
assert isclose(masked_nll(changed_action, masks), 0.6)
for bad_mask in ([1], [0] * 5, [1, 1, 2, 1, 1]):
    try:
        masked_nll(logprobs, bad_mask)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid mask accepted")
print(f"trainable_tokens={sum(masks)} total_reward={sum(rewards):.1f} masked_nll={baseline:.2f}")

trainable_tokens=4 total_reward=0.8 masked_nll=0.50

Masking the observation's loss doesn't remove it from attention or prevent gradients from depending on its context. The check changes a supplied score, not the observation tokens fed to a real model.

Read the main code paths

With the artifact in hand, code reading has an order. Start at skyrl/train/trainer.py, then follow the generator contract and the environment adapter. The trainer initializes weight-sync state, prepares the sampler, optionally evaluates, and requests a batch. Each next operation consumes the previous operation's output, so a missing field is easier to locate at the handoff where it disappears.

The generator contract lives in skyrl/train/generators/base.py. Its abstract method is asynchronous even for a synchronous RL schedule. A browser, remote verifier, or tool process can be waiting while the generator yields control. A custom generator has one required method, generate(), and returns output in input order. That order is another part of the ledger: IDs and rewards must still line up when the batch comes back.

The environment path stays outside the trainer. SkyRLGymGenerator adapts Gymnasium-style tasks, while examples also integrate verifiers, OpenEnv, Harbor, and Terminal-Bench. The agent layer adds another adapter for long-horizon loops. A new task therefore changes the edge that executes actions and computes reward, while the core still consumes the same output contract.

The trainer's order of operations

Read the regular loop as three handoffs.

First, prompts and sampling parameters enter the generator; trajectories come back with rewards, masks, IDs, and any timing metadata. Dynamic sampling and step-wise merging then decide which complete records reach the trainer.

Second, SkyRL converts those records into training input and computes current log probabilities, values when a critic is configured, advantages, and returns. Only then do policy and optional critic workers update.

Finally, checkpoints and evaluation produce evidence, and weight sync publishes the resulting policy to the next sampling window.

In the synchronous schedule, the batch's environment interactions finish before optimization, and weight sync completes before the next batch starts. Fully async deliberately relaxes that ordering: other groups can still be interacting with environments while this batch trains. Neither schedule replaces sandbox isolation, correct token alignment, or explicit version logging.

Synchronous RL: the reference schedule

Start with a schedule that separates sampling from updates. Generate a batch with policy version $\pi_k$, wait for every response and reward, update to $\pi_{k+1}$, then send that version to the sampler. The next batch starts only after every sampler crosses the sync boundary. For query-42_0, the whole SQL repair finishes before weights change. A partial or unverified transfer can still break this contract, even in synchronous mode.

Why keep this slower baseline? It gives policy-gradient math a clean reference. For a response $y$ sampled from prompt $x$, the trainer uses an objective of the form:

where $A(x,y)$ is an advantage estimate. Minimizing this objective encourages higher probability for positive-advantage actions and lower probability for negative-advantage actions. Shared parameters couple updates across actions and samples, so an individual action's probability can move either way. PPO clips the probability-ratio objective to limit incentives for large changes. The resulting policy can still move beyond the clip range because clipping does not constrain every final probability ratio.

GRPO replaces a learned value baseline with relative scores within a group of samples for the same prompt. Identical rewards give zero centered reward advantage. An enabled KL penalty or another auxiliary term can still contribute gradients, so zero reward variance doesn't universally mean zero optimizer update.

Some recipes also divide by the group standard deviation; the tiny example below uses mean centering only, matching the Dr. GRPO-style optional skip.

What synchronous training buys

Use this table as a baseline for the async choices that follow. Each benefit comes from waiting at one visible boundary; each cost is the idle time that boundary exposes.

The cost appears when tools are slow or trajectories have uneven lengths. One browser task can hold an entire step open while other GPUs wait. Synchronous mode is still the right first implementation for a new environment: it limits moving parts and gives you a trustworthy reward, mask, and version baseline before overlap enters the picture.

One-step async and fully async

The baseline exposes the tension: waiting protects freshness but lets a straggler idle the cluster. Suppose three SQL repairs finish in four seconds and query-42_0 needs forty. One-step off-policy pipelining can generate the next batch while the current one trains, but a slow generation batch still stalls its own handoff. Fully asynchronous training goes further: generation workers keep producing groups in a bounded buffer, and after each trainer step the inference client pauses in-flight requests, syncs weights, and resumes those trajectories.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

That distinction predicts the debugging surface. One-step overlap adds a pipeline boundary. Fully async adds a freshness budget, queue-capacity control, pause/resume around weight sync, and metrics that tell you how old each group was when consumed. A throughput gain without those measurements leaves the opening incident hidden.

The fully async implementation lives in fully_async_trainer.py. It creates one asyncio.Task per generation worker, places completed groups in an asyncio.Queue, and uses an AsyncStalenessManager to limit producer capacity. Here a GRPO group is n_samples_per_prompt trajectories for one prompt. Control stays within an epoch: SkyRL doesn't run generation from epoch $t+1$ while epoch $t$ is still training.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

View larger

Staleness as a budget

max_staleness_steps limits the aggregate admission budget; individual trajectories can exceed that value. Let $s_i$ be the trainer's current global step when group $i$ is consumed and $g_i$ be the step when generation was scheduled. SkyRL records:

If query-42_0 starts at step 1 and reaches the trainer at step 4, its staleness is 3. The admission controller isn't a rule that drops every group whose number exceeds $S$. With mini-batch size $B$ and max_staleness_steps $S$, at global step $s$ it keeps

accepted is cumulative across training: it includes completed groups and the current queue. At the start of step $s$, completed training has consumed $(s-1)B$ groups. Subtracting that count leaves at most $B(S+1)$ groups buffered or running. A separate worker-concurrency bound can reduce admission further. With $S=0$, only the current mini-batch fits, so generation can't get ahead of its update.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Scheduling lag also differs from exact policy provenance. A long trajectory can resume under new weights several times. Its recorded start step doesn't identify the behavior policy for every token; preserve rollout log probabilities and log sync boundaries when interpreting correction ratios.

The inequality is aggregate. With max_staleness_steps = 2, a long query-42_0 group can still arrive with staleness 3. Current behavior accepts it and logs a warning. Track the full lag distribution alongside submitted, running, accepted, and filtered counts. Frequent high-lag groups point to a producer outrunning the consumer or generation latency with a long tail.

That budget comes with configuration constraints. The docs limit fully async to generators that talk to /chat/completions. generator.batched must be false, dynamic sampling must be disabled, train_batch_size must equal policy_mini_batch_size, and colocation of all training and inference workers isn't supported. The pinned trainer checks dynamic sampling against Python None; use YAML null in the sketch below.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Instead of synchronous dynamic_sampling, fully async offers sample_full_batch with zero-variance filtering. A dropped group moves from accepted to filtered, freeing admission capacity. The invariant is submitted = accepted + filtered + running. Without that reclassification, filtered groups could fill the capacity limit and block producers forever.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Filtering changes the sample population and consumes extra prompts. If the epoch runs out before a full useful mini-batch exists, the partial batch is discarded and the epoch ends early. Trained and filtered prompt IDs are both persisted and skipped on resume. Count them separately; an all-zero-reward dataset cannot produce a useful batch under this filter.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

The filter compares the reward spread among live trajectories, meaning rows with nonzero loss masks, using a configurable tolerance. Groups with at most one live trajectory are retained by this helper; inspect sample counts and masks to confirm the admitted group contains comparative reward signal.

AReaL, DAPO, and off-policy correction

The systems question is now explicit: can language-model RL keep GPUs busy without letting policy lag erase the training signal? AReaL frames the same trade-off around in-flight generation, workload balance, and staleness control. SkyRL's manager follows that vocabulary, while its own queue and acceptance behavior still need to be read from the local revision.^{[11]Reference 11AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoninghttps://arxiv.org/abs/2505.24298}

DAPO supplies a second comparison. Its open system makes dynamic sampling and token-level policy-gradient choices part of the algorithm. SkyRL's sample_full_batch path is an async-native analogue of filter sampling. Establish experimental parity by comparing filtering, reward normalization, rollout policy versions, and evaluation protocol.^{[12]Reference 12DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

Staleness isn't the only source of off-policy behavior. Training and inference can disagree at one nominal weight version because kernels, expert routing, or parallelism differ. SkyRL's correction stack offers truncated importance sampling, geometric sequence masking, token masking, and mixture-of-experts (MoE) routing replay. These controls address residual mismatch; they don't turn an unbounded queue into fresh data.^{[13]Reference 13Off-Policy Correction at bce9ee9https://github.com/NovaSky-AI/SkyRL/blob/bce9ee9a80fbd262db44c79d5af12291ced5492d/docs/content/docs/algorithms/off_policy_correction.mdx}

Use papers as design evidence, then trace the local code that realizes each assumption. A queue can overlap work. Only the lag distribution, reward diversity, and held-out curve can show whether that overlap still produces learnable batches.

Inference and weight synchronization

The opening incident ends at the sampler. After the trainer creates $\pi_{12}$, which process receives it, and which process keeps serving $\pi_{11}$ if the transfer is partial? SkyRL separates training and sampling because their hardware and runtimes differ.

The training path commonly uses FSDP or Megatron, with JAX available through the Tinker path; generation typically reaches vLLM over HTTP. Training and generation can be colocated on one GPU set or disaggregated across workers.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The pinned commit's Linux FSDP and Megatron extras require vllm==0.28.0. Replacing the sampler with today's latest server is a compatibility change: v0.30.0, for example, changed scale-out route registration. Check the recipe's server/router dependencies and actual generation and control endpoints before combining versions. An OpenAI-compatible chat response doesn't demonstrate pause/resume or weight-transfer support.^{[16]Reference 16SkyRL Dependencies (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/pyproject.toml[17]Reference 17vLLM v0.30.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.30.0}

The official inference architecture separates ordinary token traffic from calls on the control plane. Data-plane requests need routing; control-plane calls need to reach every replica. Three pieces carry those jobs:

Generation requests travel through proxy_url: /v1/chat/completions, /v1/completions, tokenize, and related routes. Pause, resume, sleep, wake, and weight-sync endpoints fan out to every server_url. Built-in generators pass the trajectory ID as X-Session-ID, so later turns of query-42_0 stay on one replica and can reuse its prefix cache. sticky_least_loaded is the documented policy for long multi-turn work: the first turn chooses the least-loaded replica, and later turns stay there.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

View larger

Weight synchronization is where "on-policy" becomes an operational claim. In non-colocated mode, SkyRL pauses with /pause?mode=keep, freezes in-flight rollouts with KV state preserved, broadcasts tensors over NCCL from trainer rank 0, then resumes.

Current inference docs spell out a four-stage transfer: /init_weight_transfer_engine, /start_weight_update, chunked /update_weights, then /finish_weight_update. Sending a chunk is different from finishing the update; keep generation paused until the selected protocol completes.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

In colocated mode it uses CUDA IPC handles plus /sleep and /wake_up so the inference engine can free VRAM during the train step. Project lineage and older docs also describe Gloo and checkpoint-and-load paths; verify the strategy at the revision you deploy. No path removes the need to record policy versions.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

The current inference guide also documents delta transfers through a shared filesystem or object store and sharded_rdt transfers pulled through NIXL/RDMA. These have different storage and network requirements from NCCL and IPC. Select and test the backend your deployment actually constructs.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

Fully async adds a KV-cache decision. clear_kv_cache_on_weight_sync defaults to false, so a resumed trajectory can retain KV state computed with older weights. generator.use_cache_salt separates prefix-cache reuse across versioned requests; it doesn't make an already-running trajectory fresh. Clearing KV state forces recomputation, but previously sampled actions and buffered groups remain off-policy. It doesn't restore the synchronous sampling schedule.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

A sync failure path

Return to step 12. One replica still serves step 11 weights, so routing can give the next group a mixed policy version. Treat that as an evidence problem first, not as a reward problem. A safe recovery sequence is:

1. Pause new generation if the endpoint supports it.
2. Record trainer step and sampler version for every replica.
3. Check transfer logs and replica health.
4. Retry or roll back the sync operation.
5. Resume only when all replicas report the expected version.

This operator acceptance contract records the client counter as one signal. Collect sampler-side version evidence for every replica when the client view lacks that coverage. Quarantine a replica with uncertain weights until its sampling probabilities are known. Validate the full update boundary after the RPC succeeds.

Tinker: a stable API over changing hardware

What if the experiment should keep its training calls while the hardware changes underneath? Tinker is a training API from Thinking Machines Lab that presents training and sampling through a service-like interface. SkyRL implements a Tinker-compatible backend so the same style of program can run on local hardware. The unified package carries that work under skyrl; the old skyrl-tx directory records the migration.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The official architecture divides the service into three layers:

The lifecycle is easy to test with one update. A user submits an operation and receives a future-like handle. Training calls (forward_backward, optim_step, forward) travel through the engine to GPU workers, while sampling wraps RemoteInferenceClient.

After optim_step, the client must call save_weights_for_sampler() before the next sample if it wants the new policy. Persistent mode also writes a Hugging Face checkpoint; ephemeral mode syncs weights and returns a sampling client without the disk write, which suits a hot RL loop that syncs every batch.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

Use save_weights_and_get_sampling_client() for that ephemeral path. A persistent sampler export still isn't the full training-resume artifact: save_checkpoint() preserves model weights, optimizer state, and the learning-rate scheduler. Complete the corresponding operation before relying on its result.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The Tinker architecture guide checked on October 10, 2026 identifies cross_entropy and importance_sampling as validated API losses. Native-trainer registry names such as gspo, sapo, and cispo are not yet wired through that conversion path: the batch lacks the required advantages and old log probabilities. Changing the loss-name string cannot supply those fields. Check the converted batch and an executed update before porting a native-trainer recipe to Tinker.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The boundary can drift. Backend-specific loss support, normalization, checkpoint formats, and model constraints still need integration tests. Don't infer API compatibility merely from a shared loss name. Pin the SkyRL commit, SDK, backend versions, tokenizer, and recipe configuration before comparing runs.

SkyRL-Agent: long-horizon work belongs above the trainer

The same ledger gets longer for SWE (software-engineering) agents, web researchers, and terminal users. One episode may make many tool calls, carry state across turns, and fail because a test, shell, or network service broke. SkyRL-Agent puts task logic, tools, dispatch strategies, and training-backend adapters above the trainer. It can connect to OpenAI-compatible serving such as vLLM, veRL, SkyRL-Train, or Tinker through configuration.^{[4]Reference 4SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

The paper's systems claim is specific. An optimized asynchronous pipeline dispatcher reported a $1.55\times$ speedup over naive asynchronous batching by overlapping CPU-bound tool work with GPU generation.

Using that stack, the authors trained SA-SWE-32B from Qwen3-32B with pure RL. The paper reports the base at 24.4% Pass@1 and SA-SWE-32B at 39.4% Pass@1 on SWE-Bench Verified, under a simplified ReAct loop with file-editor and bash tools, 40k context, and 100 max steps. That's a dated paper snapshot, not a live leaderboard. Copy the protocol before comparing the numbers.^{[4]Reference 4SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

The agent layer keeps environment-specific logic close to the task. Browser tools return page observations, code-execution tools return stdout and process status, and finish tools mark episodes complete. The dispatcher can run work asynchronously, while the training backend consumes a normalized trajectory. That division makes a failed test diagnosable without pretending it was a bad gradient.

Application map

The same library can support all six, but the reward contract isn't interchangeable. SQL execution may yield an exact reward; research agents may combine evidence coverage, citation validity, and answer quality; terminal tasks may depend on flaky or expensive tests. Keep those semantics documented next to the environment.

A worked policy update

For a separate binary-reward SQL experiment, sample two queries for the same prompt. A frozen result checker gives the correct query reward 1 and the incorrect query reward 0. Their mean is $0.5$, giving centered advantages $+0.5$ and $-0.5$. This binary reward deliberately differs from the earlier shaped turn rewards. Choose the reward contract for the experiment and document how it maps into advantages and loss masks.

With $y_+$ denoting the correct query and $y_-$ the incorrect one, the simplified loss encourages that change:

That equation hides three implementation details:

1. The group must contain comparable samples for the same prompt.
2. The reward must be aligned with the environment's true success condition.
3. The loss mask must identify the response tokens that policy optimization should change.

If both responses receive reward 1, the relative advantages are zero. That can mean an easy prompt or a broken evaluator. Inspect which case applies before changing the learning rate; more samples don't repair a checker that always returns success.

The standard-library example checks mean-centered rewards and the exact counter trace in the figure. It's a small arithmetic model, not a SkyRL training run. Each row is taken just before consumption; accepted counts include groups used by earlier updates.

from math import isfinite
from statistics import fmean

def group_advantages(rewards: list[float]) -> tuple[list[float], bool]:
    if not rewards or not all(isfinite(r) for r in rewards):
        raise ValueError("rewards must be nonempty and finite")
    baseline = fmean(rewards)
    advantages = [reward - baseline for reward in rewards]
    zero_variance = len(set(rewards)) == 1
    return advantages, zero_variance

sql_group = [1.0, 0.0, 1.0, 0.0]
advantages, zero_variance = group_advantages(sql_group)
assert advantages == [0.5, -0.5, 0.5, -0.5]
assert zero_variance is False
assert group_advantages([1.0, 1.0, 1.0, 1.0])[1] is True

print("advantages", advantages)
# (step, consumed group, scheduled step, accepted, running)
trace = [(1, "A", 1, 2, 1), (2, "B", 1, 3, 1),
         (3, "C", 2, 3, 1), (4, "L", 1, 4, 0)]
B, S = 1, 2
for step, group, scheduled, accepted, running in trace:
    consumed = (step - 1) * B
    outstanding = accepted - consumed + running
    assert accepted + running <= (S + step) * B
    assert 0 <= outstanding <= B * (S + 1)
    print(f"step={step} group={group} lag={step-scheduled} outstanding={outstanding}")
assert trace[-1][0] - trace[-1][2] > S

for invalid in ([], [float("nan")], [float("inf")]):
    try:
        group_advantages(invalid)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid reward accepted")

advantages [0.5, -0.5, 0.5, -0.5]
step=1 group=A lag=0 outstanding=3
step=2 group=B lag=1 outstanding=3
step=3 group=C lag=1 outstanding=2
step=4 group=L lag=3 outstanding=1

The long group L violates the nominal two-step lag budget without violating aggregate capacity. The horizontal axis counts consumption steps. Measure throughput and lag under load to estimate performance for this schedule.

The configuration sketch below exposes the decisions without pretending to be a complete launch recipe. It uses names from SkyRL's documented config model.

trainer:
  train_batch_size: 1
  policy_mini_batch_size: 1
  algorithm:
    advantage_estimator: grpo
    policy_loss_type: rollout_is
    zero_variance_filter: true
    dynamic_sampling:
      type: null
  placement:
    colocate_all: false
  fully_async:
    enabled: true
    max_staleness_steps: 2
    num_parallel_generation_workers: 3
    sample_full_batch: true
    clear_kv_cache_on_weight_sync: false
generator:
  batched: false
  n_samples_per_prompt: 4
  use_cache_salt: true
  inference_engine:
    backend: vllm
environment:
  env_class: text2sql

Use the FullyAsyncRayPPOTrainer entrypoint and a revision-matched HTTP generator. rollout_is uses behavior-policy log probabilities; configure the generator to return them. YAML null becomes Python None, unlike the literal string none still shown in some documentation. This sketch omits the model, tokenizer, dataset, placement, and CUDA environment, so it isn't a launchable configuration.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

Check the pinned implementation without a GPU

The downloadable contract probe checks the upstream source bytes against fixed SHA-256 hashes, then executes only the reviewed scheduling classes, buffer-drain method, and reward-stamping method. It avoids the GPU-dependent module imports. Run it from the article directory; --fetch downloads the two public source files into a temporary source directory.

uv run assets/verify_pinned_contracts.py --fetch

The recorded CPU receipt contains eight passing checks, including a blocked producer released by filtering, epoch accounting, checkpoint capacity restoration, the long-group trace, buffer exhaustion, and reward truncation/EOS branches. Those are executed method checks, not a full upstream test-suite pass. A custom chat-template branch returns the final scalar reward instead of the per-token vector, so a generator extension must test its selected path.^{[8]Reference 8SkyRL Gym Generator (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/skyrl_gym_generator.py}

No CUDA model, Ray cluster, vLLM HTTP server, real SQL environment, weight transfer, or learning curve was executed for this review. The simulated trainer still needs inference endpoints and skips actual optimization and broadcast, so distributed updates remain untested. Before a real experiment, run the revision's small synchronous recipe, verify a save/resume round trip, then compare async against that baseline at equal evaluated prompts and GPU-hours.

Strengths and weaknesses

A component count doesn't say whether query-42_0 stayed aligned. A run does. The first table asks what each seam buys. The evidence column names the measurement that can confirm it.

Strengths

The useful sequence is incremental: start with synchronous GRPO, add a custom generator, then test async generation while preserving output fields. Each experiment changes one boundary at a time, so an ablation can point to a seam instead of a pile of infrastructure changes.

Weaknesses and sharp edges

These edges define the admission bar for a serious RL experiment: reproducible environment state, traceable policy versions, bounded queues, and an explicit stop when rewards or sync health become untrustworthy. The operator section turns that bar into evidence you can collect before burning more GPU hours.

Research papers and design lineage

The core path is now complete. Keep the papers as a second reading layer: they explain why a workload matters and what was measured, while the code shows which assumptions became interfaces and which remain configuration constraints.

This table supplies research context rather than a leaderboard. DAPO and AReaL cover related systems, while SkyRL provides a framework for testing environment and execution-plan variants. Compare exact task, base model, reward, sampling, and update settings before importing a result.

Admit a run and recover it

Before launching a serious SkyRL run, write its evidence contract. If one answer is unknown, use a smaller smoke job before spending more GPU hours. The order matters: trust the reward, identify the policy, bound overlap, then rehearse recovery.

Gate 1: make reward traceable

Define one episode and its termination condition. Record which tokens receive policy loss and whether reward is scalar, per-turn, or per-token. Then ask whether the evaluator can give every sample the same score and whether tool outputs are reproducible under a fixed seed and environment snapshot. A zero-variance group or nondeterministic tool can flatten a run before optimization gets a chance to help.

Gate 2: name every weight owner

Write down which process owns training, reference, and sampler weights, whether training and generation are colocated or disaggregated, and which inference and weight-sync backends are selected. After every sync, require every serving replica to report the same policy version. Pin model, tokenizer, CUDA, Ray, and dependency versions so a replay has a meaningful environment.

Gate 3: bound async work

Set mini-batch size, generation-worker count, and max_staleness_steps together. Calculate completed-group plus in-flight headroom as $B(S+1)$, choose the pause mode (keep, abort, or wait), and check whether use_cache_salt isolates prefix-cache blocks across policy versions. Decide which trajectories are filtered as lag rises, how generation failures surface, and which metric stops the run automatically.

Gate 4: keep a recovery packet

A replayable trajectory joins prompt, actions, observations, rewards, and policy version. Logs separate environment latency from inference latency. Require checkpoint saves to be atomic and resumable, and keep a rollback path for reward or sync regressions. Finally, name the offline slice that must pass before online gains count as learning.

🎯 Production tip: Log trajectory ID, scheduled policy step, consumed policy step, reward components, stop reason, and environment revision together. A throughput chart without those joins fails to explain a bad update.

When the extra seams pay off

Choose this stack when the research question includes an environment, a changing rollout policy, or a nontrivial training and inference plan. Tool-use agents, verifier-backed reasoning, Text-to-SQL, terminal tasks, and sync-versus-async experiments all cross those boundaries.

A smaller trainer fits a static supervised dataset with no environment or rollout control. SkyRL's interfaces add coordination cost, so their value starts when one of those boundaries is part of the question.

There is a second fork when policy or data must stay outside your GPU cluster. SkyRL-Tinker offers a service-like API with local hardware control, but its compatibility boundary still needs a pinned integration test.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The practical handoff is hybrid: prototype reward and environment logic in a synchronous local run, validate trajectory fields and evaluator slices, then move to disaggregated or fully async execution after a baseline and staleness budget exist. That order lets a later throughput win answer a useful question: did the system learn more, or did it only move more stale tokens?

Keep query-42_0 aligned from the observation mask, through the staleness counters, to the replica that still serves the previous step. If a chart can't do that, it isn't a learning result.

Review an extension before running it

Evaluation rubric

• Foundational: Trace one response through token IDs, observation masks, reward placement, and the final stop reason without treating a tool observation as a sampled action.
• Intermediate: Reconstruct accepted, running, filtered, and consumed counts, including a long group whose scheduling lag exceeds the aggregate budget.
• Advanced: Separate backend/API compatibility, sampler-version evidence, cache behavior, and held-out learning outcomes when accepting an async change.

Follow-up questions