Deep Dive - SkyRL

A step-12 SQL-repair run can hit this boundary: a trajectory is still waiting on its database when training finishes an update. Two serving replicas load the new weights; a third answers the next turn with the old ones. The query can still earn a good reward, yet the run no longer knows which model version produced each action. That policy-version boundary is the incident that gives SkyRL its shape.

Follow one trajectory, query-42_0, from prompt through generator, environment, trainer, and weight sync. Its token IDs, loss masks, reward, and identity must survive every handoff. Version tracking needs extra care: a scheduled trainer step is recorded for each async group, but that number isn't a per-token record of the weights used during an in-flight update.

The implementation walkthrough is pinned to official SkyRL commit 955c3e23bf8939b28a9d6c424308a762dcb8a2e7, checked on September 2, 2026. It uses the unified skyrl/ package; skyrl-gym and skyrl-agent remain separate directories. Older skyrl-train and skyrl-tx recipes use different paths. Match a recipe to its commit before copying commands.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Replaceable interfaces, not one trainer

Suppose query-42_0 becomes a terminal task tomorrow. Which code should change? The environment may change from SQL execution to shell commands, and the generator may change how it parses turns. Advantage calculation should keep receiving the same trajectory contract. That is the boundary to look for before opening a trainer file.

SkyRL names four working seams: trajectory generator, environment, inference engine, and trainer. A controller role also owns placement, initialization, and control flow, although current training control still lives in trainer.py while extraction of a standalone controller remains work in progress. Explicit inputs and outputs keep the trainer from knowing every task implementation.^{[2]Reference 2SkyRL Overviewhttps://docs.skyrl.ai/docs/getting-started/overview}

This makes an extension local. A Text-to-SQL task adds an environment or generator; it doesn't edit advantage calculation. A local vLLM worker can be replaced by a remote OpenAI-compatible endpoint without moving reward processing. One-step pipelining can become fully asynchronous while the trajectory fields stay recognizable. The schedule and freshness policy change, but the handoff still answers the same ledger questions.

The native trainer includes Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and several policy-gradient variants. Its documented fully async backends are Fully Sharded Data Parallel (FSDP) and Megatron. The Tinker path also includes JAX; backend presence doesn't imply every trainer supports it. HTTP inference uses vLLM API servers plus a router.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

💡 Key insight: If a seam preserves prompt, token, reward, mask, and version semantics, the implementation behind it can change without changing the learning loop.

Project identity and lineage

Before reading implementation details, separate project provenance from system ownership. SkyRL is developed at the Berkeley Sky Computing Lab in collaboration with Anyscale. Its repository acknowledges compute support from Databricks, NVIDIA, Lambda Labs, AMD, AWS, Modal, and Daytona. Those acknowledgments explain context; they don't assign every subsystem to every organization.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

The lineage is visible in the repository layout and release notes:

The project also credits ideas and code from veRL, OpenRLHF, Search-R1, OpenReasonerZero, and NeMo-RL. Keep that lineage in the background while tracing one batch. It maps the ecosystem, but it doesn't make SkyRL a drop-in replacement for any one project.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Generator, environment, trainer

Return to query-42_0. Before naming classes, predict what the trainer needs after two SQL turns: the sampled response, the database observation, the reward, and a way to distinguish trainable tokens from context. If that record is complete, the optimizer can stay unaware of whether the environment is SQL, a terminal, or a browser.

A controller role sets up execution. A Generator coordinates inference and environments to produce trajectories. A Trainer turns those trajectories into gradients. Current control flow remains in trainer.py, while InferenceEngine and Environment stay replaceable boundaries inside the Generator.

The loop isn't "model in, loss out." A multi-turn trajectory can include token IDs, tool calls, observations, rewards, per-token masks, and a stop reason. Those fields carry the decision from environment execution into optimization. Without them, a trainer can apply loss to an observation, credit a failed tool call as success, or join a retry to the wrong episode.

Vocabulary at the seams

The table is a decoder ring for the code. Read Rollout as the artifact being handed forward, and read the final column as the bug that appears when one boundary silently changes its meaning.

GeneratorInput includes prompts, environment classes, optional environment extras, sampling parameters, trajectory IDs, and batch metadata. GeneratorOutput returns batched prompt and response token IDs, rewards, loss masks, stop reasons, rollout metrics, optional log probabilities, trajectory IDs, and timing splits. It also includes step-wise completion flags and vision tensors. TypedDict describes the interface for type checkers; it doesn't validate a custom generator's arrays at runtime.^{[7]Reference 7SkyRL Generator Contract (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/base.py}

The architecture in one request

The same handoff must work for a one-turn math answer and a long tool-use task. A generator may call an environment many times before returning one output, but the trainer still receives one normalized object. That normalization is what lets us reuse the rest of the route for query-42_0.

The two feedback paths have different jobs: inference samples actions, while the environment executes them and returns observations. The trainer receives completed records rather than raw tool traffic. Zero-variance rewards, broken tools, stale groups, and failed transfers need separate handling at those boundaries.

Why does SkyRL put loss_masks in GeneratorOutput instead of asking the trainer to infer them from text?

Answer

The generator knows which tokens are model actions, tool arguments, observations, or padding. Carrying an explicit mask prevents the trainer from learning from tokens that should be context only, especially in multi-turn or step-wise trajectories.

A concrete trajectory

Now follow the ledger instead of the class diagram. The prompt is: "List active customers with more than three unresolved incidents." The generator asks for a SQL action, sends it to a sandboxed database environment, appends either an error or rows as an observation, and gives the model another turn to repair its query. The environment supplies evidence; the loss mask decides which tokens can move the policy.

Call this trajectory query-42_0. Should the database error receive policy loss? No: it's context, not a sampled action. To keep the alignment visible, use a toy vocabulary with two tokens for an attempted query, one observation token, and two tokens for its repair. These IDs don't come from a real tokenizer, and the labels aren't executable SQL. This is one row of a batched output, with optional fields omitted:

The reward field accepts either one scalar per trajectory or one reward per response token. It isn't an arbitrary list of turn rewards: the default multi-turn generator stamps each turn reward onto the corresponding final action token. Here the total is 0.8; aggregation and advantage estimation decide how that return influences the sampled actions.^{[7]Reference 7SkyRL Generator Contract (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/base.py[8]Reference 8SkyRL Gym Generator (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/skyrl_gym_generator.py}

The IDs and masks aren't decoration. SkyRLGymGenerator enforces token-in-token-out in its default multi-turn path: it appends sampled token IDs, appends tokenized observations with mask 0, then samples again. If you retokenize the whole chat from strings, token boundaries can change after the sampler has finished. Training then sees a different sequence from the one that earned the reward.^{[9]Reference 9SkyRL-Gym Generatorhttps://docs.skyrl.ai/docs/tutorials/skyrl_gym_generator}

A multi-turn batch can flatten several conversations. Step-wise training can return one turn at a time and mark whether an episode is complete. Keep those distinctions attached to each row, or a retry and its reward can be joined to the wrong turn.

The toy row below checks alignment and computes a masked mean negative log probability. This isn't the GRPO loss; it isolates what the mask does. Changing the observation's supplied score doesn't change this objective, while changing a sampled action's score does.

Masking the observation's loss doesn't remove it from attention or prevent gradients from depending on its context. The check changes a supplied score, not the observation tokens fed to a real model.

Read the main code paths

With the artifact in hand, code reading has an order. Start at skyrl/train/trainer.py, then follow the generator contract and the environment adapter. The trainer initializes weight-sync state, prepares the sampler, optionally evaluates, and requests a batch. Each next operation consumes the previous operation's output, so a missing field is easier to locate at the handoff where it disappears.

The generator contract lives in skyrl/train/generators/base.py. Its abstract method is asynchronous even for a synchronous RL schedule. A browser, remote verifier, or tool process can be waiting while the generator yields control. A custom generator has one required method, generate(), and returns output in input order. That order is another part of the ledger: IDs and rewards must still line up when the batch comes back.

The environment path stays outside the trainer. SkyRLGymGenerator adapts Gymnasium-style tasks, while examples also integrate verifiers, OpenEnv, Harbor, and Terminal-Bench. The agent layer adds another adapter for long-horizon loops. A new task therefore changes the edge that executes actions and computes reward, while the core still consumes the same output contract.

The trainer's order of operations

Read the regular loop as three handoffs.

First, prompts and sampling parameters enter the generator; trajectories come back with rewards, masks, IDs, and any timing metadata. Dynamic sampling and step-wise merging then decide which complete records reach the trainer.

Second, SkyRL converts those records into training input and computes current log probabilities, values when a critic is configured, advantages, and returns. Only then do policy and optional critic workers update.

Finally, checkpoints and evaluation produce evidence, and weight sync publishes the resulting policy to the next sampling window.

In the synchronous schedule, the batch's environment interactions finish before optimization, and weight sync completes before the next batch starts. Fully async deliberately relaxes that ordering: other groups can still be interacting with environments while this batch trains. Neither schedule replaces sandbox isolation, correct token alignment, or explicit version logging.

Synchronous RL: the reference schedule

Start with a schedule that separates sampling from updates. Generate a batch with policy version $\pi_k$, wait for every response and reward, update to $\pi_{k+1}$, then send that version to the sampler. The next batch starts only after every sampler crosses the sync boundary. For query-42_0, the whole SQL repair finishes before weights change. A partial or unverified transfer can still break this contract, even in synchronous mode.

Why keep this slower baseline? It gives policy-gradient math a clean reference. For a response $y$ sampled from prompt $x$, the trainer uses an objective of the form:

$$L(\theta) = -\mathbb{E}_{(x,y) \sim \pi_k}\left[A(x,y) \log \pi_\theta(y \mid x)\right],$$

where $A(x,y)$ is an advantage estimate. Minimizing this objective encourages higher probability for positive-advantage actions and lower probability for negative-advantage actions. Shared parameters and competing samples mean this isn't a guarantee about every individual probability. PPO clips a probability-ratio objective to limit its incentive for large changes; clipping isn't a hard bound on the resulting policy change.

GRPO replaces a learned value baseline with relative scores within a group of samples for the same prompt. Identical rewards give zero centered reward advantage. An enabled KL penalty or another auxiliary term can still contribute gradients, so zero reward variance doesn't universally mean zero optimizer update.

Some recipes also divide by the group standard deviation; the tiny example below uses mean centering only, matching the Dr. GRPO-style optional skip.

What synchronous training buys

Use this table as a baseline for the async choices that follow. Each benefit comes from waiting at one visible boundary; each cost is the idle time that boundary exposes.

The cost appears when tools are slow or trajectories have uneven lengths. One browser task can hold an entire step open while other GPUs wait. Synchronous mode is still the right first implementation for a new environment: it limits moving parts and gives you a trustworthy reward, mask, and version baseline before overlap enters the picture.

You add a browser environment and the first synchronous run sometimes gives a reward of zero for every sample. What should you inspect before changing the optimizer?

Answer

Inspect environment termination, reward calculation, and generated action parsing first. A zero-variance group produces no useful relative signal in GRPO, and a malformed tool action can make every sample look equally wrong even when the policy is improving.

One-step async and fully async

The baseline exposes the tension: waiting protects freshness but lets a straggler idle the cluster. Suppose three SQL repairs finish in four seconds and query-42_0 needs forty. One-step off-policy pipelining can generate the next batch while the current one trains, but a slow generation batch still stalls its own handoff. Fully asynchronous training goes further: generation workers keep producing groups in a bounded buffer, and after each trainer step the inference client pauses in-flight requests, syncs weights, and resumes those trajectories.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

That distinction predicts the debugging surface. One-step overlap adds a pipeline boundary. Fully async adds a freshness budget, queue-capacity control, pause/resume around weight sync, and metrics that tell you how old each group was when consumed. A throughput gain without those measurements leaves the opening incident hidden.

The fully async implementation lives in fully_async_trainer.py. It creates one asyncio.Task per generation worker, places completed groups in an asyncio.Queue, and uses an AsyncStalenessManager to limit producer capacity. Here a GRPO group is n_samples_per_prompt trajectories for one prompt. Control stays within an epoch: SkyRL doesn't run generation from epoch $t+1$ while epoch $t$ is still training.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

Staleness as a budget

Treat max_staleness_steps as an aggregate budget, not a promise about every individual trajectory. Let $s_i$ be the trainer's current global step when group $i$ is consumed and $g_i$ be the step when generation was scheduled. SkyRL records:

$$\operatorname{staleness}_i = s_i - g_i.$$

If query-42_0 starts at step 1 and reaches the trainer at step 4, its staleness is 3. The admission controller isn't a rule that drops every group whose number exceeds $S$. With mini-batch size $B$ and max_staleness_steps $S$, at global step $s$ it keeps

$$\text{accepted} + \text{running} \le (S + s)\,B.$$

accepted is cumulative across training: it includes already-trained groups, not just the queue. At the start of step $s$, completed training has consumed $(s-1)B$ groups. Subtracting that count leaves at most $B(S+1)$ groups buffered or running. A separate worker-concurrency bound can reduce admission further. With $S=0$, only the current mini-batch fits, so generation can't get ahead of its update.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Scheduling lag also differs from exact policy provenance. A long trajectory can resume under new weights several times. Its recorded start step doesn't identify the behavior policy for every token; preserve rollout log probabilities and log sync boundaries when interpreting correction ratios.

The inequality is aggregate. With max_staleness_steps = 2, a long query-42_0 group can still arrive with staleness 3. Current behavior accepts it and logs a warning. Track the full lag distribution alongside submitted, running, accepted, and filtered counts. Frequent high-lag groups point to a producer outrunning the consumer or generation latency with a long tail.

That budget comes with configuration constraints. The docs limit fully async to generators that talk to /chat/completions. generator.batched must be false, dynamic sampling must be disabled, train_batch_size must equal policy_mini_batch_size, and colocation of all training and inference workers isn't supported. The pinned trainer checks dynamic sampling against Python None; use YAML null in the sketch below.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Instead of synchronous dynamic_sampling, fully async offers sample_full_batch with zero-variance filtering. A dropped group moves from accepted to filtered, freeing admission capacity. The invariant is submitted = accepted + filtered + running. Without that reclassification, filtered groups could fill the capacity limit and block producers forever.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

Filtering changes the sample population and consumes extra prompts. If the epoch runs out before a full useful mini-batch exists, the partial batch is discarded and the epoch ends early. Trained and filtered prompt IDs are both persisted and skipped on resume. Count them separately; sample_full_batch doesn't guarantee a useful batch from an all-zero-reward dataset.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py}

The filter compares the reward spread among live trajectories, meaning rows with nonzero loss masks, using a configurable tolerance. Groups with at most one live trajectory are retained by this helper, so admission alone doesn't prove that a group contains comparative reward signal. Validate sample count and masks before interpreting the accepted rate.

Why is "more generation workers" not an unconditional fix for a slow fully async run?

Answer

More workers can fill the buffer faster, but they can also push staleness beyond the budget, increase memory pressure, and make policy updates consume older data. Tune worker count together with mini-batch size, max_staleness_steps, queue capacity, and measured generation latency. Official docs also cap useful workers at policy_mini_batch_size * (max_staleness_steps + 1).

AReaL, DAPO, and off-policy correction

The systems question is now explicit: can language-model RL keep GPUs busy without letting policy lag erase the training signal? AReaL frames the same trade-off around in-flight generation, workload balance, and staleness control. SkyRL's manager follows that vocabulary, while its own queue and acceptance behavior still need to be read from the local revision.^{[11]Reference 11AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoninghttps://arxiv.org/abs/2505.24298}

DAPO supplies a second comparison. Its open system makes dynamic sampling and token-level policy-gradient choices part of the algorithm. SkyRL's sample_full_batch path is an async-native analogue of filter sampling, not evidence that the two experiments match. Compare filtering, reward normalization, rollout policy versions, and evaluation protocol before claiming parity.^{[12]Reference 12DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

Staleness isn't the only source of off-policy behavior. Training and inference can disagree at one nominal weight version because kernels, expert routing, or parallelism differ. SkyRL's correction stack offers truncated importance sampling, geometric sequence masking, token masking, and mixture-of-experts (MoE) routing replay. These controls address residual mismatch; they don't turn an unbounded queue into fresh data.^{[13]Reference 13Off-Policy Correction at bce9ee9https://github.com/NovaSky-AI/SkyRL/blob/bce9ee9a80fbd262db44c79d5af12291ced5492d/docs/content/docs/algorithms/off_policy_correction.mdx}

Use papers as design evidence, then trace the local code that realizes each assumption. A queue can overlap work. Only the lag distribution, reward diversity, and held-out curve can show whether that overlap still produces learnable batches.

Inference and weight synchronization

The opening incident ends at the sampler. After the trainer creates $\pi_{12}$, which process receives it, and which process keeps serving $\pi_{11}$ if the transfer is partial? SkyRL separates training and sampling because their hardware and runtimes differ.

The training path commonly uses FSDP or Megatron, with JAX available through the Tinker path; generation typically reaches vLLM over HTTP. Training and generation can be colocated on one GPU set or disaggregated across workers.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The official inference architecture separates ordinary token traffic from calls on the control plane. Data-plane requests need routing; control-plane calls need to reach every replica. Three pieces carry those jobs:

Generation requests travel through proxy_url: /v1/chat/completions, /v1/completions, tokenize, and related routes. Pause, resume, sleep, wake, and weight-sync endpoints fan out to every server_url. Built-in generators pass the trajectory ID as X-Session-ID, so later turns of query-42_0 stay on one replica and can reuse its prefix cache. sticky_least_loaded is the documented policy for long multi-turn work: the first turn chooses the least-loaded replica, and later turns stay there.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

Weight synchronization is where "on-policy" becomes an operational claim. In non-colocated mode, SkyRL pauses with /pause?mode=keep, freezes in-flight rollouts with KV state preserved, broadcasts tensors over NCCL from trainer rank 0, then resumes.

In colocated mode it uses CUDA IPC handles plus /sleep and /wake_up so the inference engine can free VRAM during the train step. Project lineage and older docs also describe Gloo and checkpoint-and-load paths; verify the strategy at the revision you deploy. No path removes the need to record policy versions.^{[14]Reference 14SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

Fully async adds a KV-cache decision. clear_kv_cache_on_weight_sync defaults to false, so a resumed trajectory can retain KV state computed with older weights. generator.use_cache_salt separates prefix-cache reuse across versioned requests; it doesn't make an already-running trajectory fresh. Clearing KV state forces recomputation, but previously sampled actions and buffered groups remain off-policy. It doesn't restore the synchronous sampling schedule.^{[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

A sync failure path

Return to step 12. One replica still serves step 11 weights, so routing can give the next group a mixed policy version. Treat that as an evidence problem first, not as a reward problem. A safe recovery sequence is:

1. Pause new generation if the endpoint supports it.
2. Record trainer step and sampler version for every replica.
3. Check transfer logs and replica health.
4. Retry or roll back the sync operation.
5. Resume only when all replicas report the expected version.

This is an operator acceptance contract, not a claim that SkyRL's client counter attests every replica. Add sampler-side version evidence if your deployment doesn't expose it. Quarantine a replica with uncertain weights: an off-policy correction needs known sampling probabilities, not merely a label saying "old." A successful response alone doesn't prove a valid update boundary.

What evidence proves that a weight sync worked?

Answer

A successful RPC is not enough. Check the expected policy version on every serving replica, observe a post-sync request, confirm no in-flight request crossed an invalid pause boundary, and compare sync duration and generation error metrics with the release budget.

Tinker: a stable API over changing hardware

What if the experiment should keep its training calls while the hardware changes underneath? Tinker is a training API from Thinking Machines Lab that presents training and sampling through a service-like interface. SkyRL implements a Tinker-compatible backend so the same style of program can run on local hardware. The unified package carries that work under skyrl; the old skyrl-tx directory records the migration.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The official architecture is three layers, not a claim that every backend looks identical:

The lifecycle is easy to test with one update. A user submits an operation and receives a future-like handle. Training calls (forward_backward, optim_step, forward) travel through the engine to GPU workers, while sampling wraps RemoteInferenceClient.

After optim_step, the client must call save_weights_for_sampler() before the next sample if it wants the new policy. Persistent mode also writes a Hugging Face checkpoint; ephemeral mode syncs weights and returns a sampling client without the disk write, which suits a hot RL loop that syncs every batch.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The boundary can drift. Backend-specific loss support, normalization, checkpoint formats, and model constraints still need integration tests. Don't infer API compatibility merely from a shared loss name. Pin the SkyRL commit, SDK, backend versions, tokenizer, and recipe configuration before comparing runs.

SkyRL-Agent: long-horizon work belongs above the trainer

The same ledger gets longer for SWE (software-engineering) agents, web researchers, and terminal users. One episode may make many tool calls, carry state across turns, and fail because a test, shell, or network service broke. SkyRL-Agent puts task logic, tools, dispatch strategies, and training-backend adapters above the trainer. It can connect to OpenAI-compatible serving such as vLLM, veRL, SkyRL-Train, or Tinker through configuration.^{[4]Reference 4SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

The paper's systems claim is specific. An optimized asynchronous pipeline dispatcher reported a $1.55\times$ speedup over naive asynchronous batching by overlapping CPU-bound tool work with GPU generation.

Using that stack, the authors trained SA-SWE-32B from Qwen3-32B with pure RL. The paper reports the base at 24.4% Pass@1 and SA-SWE-32B at 39.4% Pass@1 on SWE-Bench Verified, under a simplified ReAct loop with file-editor and bash tools, 40k context, and 100 max steps. That's a dated paper snapshot, not a live leaderboard. Copy the protocol before comparing the numbers.^{[4]Reference 4SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

The agent layer keeps environment-specific logic close to the task. Browser tools return page observations, code-execution tools return stdout and process status, and finish tools mark episodes complete. The dispatcher can run work asynchronously, while the training backend consumes a normalized trajectory. That division makes a failed test diagnosable without pretending it was a bad gradient.

Application map

The same library can support all six, but the reward contract isn't interchangeable. SQL execution may yield an exact reward; research agents may combine evidence coverage, citation validity, and answer quality; terminal tasks may depend on flaky or expensive tests. Keep those semantics documented next to the environment.

A worked policy update

For a separate binary-reward SQL experiment, sample two queries for the same prompt. A frozen result checker gives the correct query reward 1 and the incorrect query reward 0. Their mean is $0.5$, giving centered advantages $+0.5$ and $-0.5$. This binary reward deliberately differs from the earlier shaped turn rewards: reward design is part of the experiment, not a universal SkyRL convention.

With $y_+$ denoting the correct query and $y_-$ the incorrect one, the simplified loss encourages that change:

$$L = -\frac{1}{2}\left[(+0.5)\log \pi_\theta(y_+\mid x) + (-0.5)\log \pi_\theta(y_-\mid x)\right].$$

That equation hides three implementation details:

1. The group must contain comparable samples for the same prompt.
2. The reward must be aligned with the environment's true success condition.
3. The loss mask must identify the response tokens that policy optimization should change.

If both responses receive reward 1, the relative advantages are zero. That can mean an easy prompt or a broken evaluator. Inspect which case applies before changing the learning rate; more samples don't repair a checker that always returns success.

The standard-library example checks mean-centered rewards and the exact counter trace in the figure. It's a small arithmetic model, not a SkyRL training run. Each row is taken just before consumption; accepted counts include groups used by earlier updates.

The long group L violates the nominal two-step lag budget without violating aggregate capacity. The figure's horizontal axis is consumption step, not time: this trace establishes a legal schedule, not a measured speedup.

The configuration sketch below exposes the decisions without pretending to be a complete launch recipe. It uses names from SkyRL's documented config model.

Use the FullyAsyncRayPPOTrainer entrypoint and a revision-matched HTTP generator. rollout_is uses behavior-policy log probabilities; configure the generator to return them. YAML null becomes Python None, unlike the literal string none still shown in some documentation. This sketch omits the model, tokenizer, dataset, placement, and CUDA environment, so it isn't a launchable configuration.^{[10]Reference 10SkyRL Fully Asynchronous Trainer (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/fully_async_trainer.py[3]Reference 3Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

Check the pinned implementation without a GPU

The downloadable contract probe checks the upstream source bytes against fixed SHA-256 hashes, then executes only the reviewed scheduling classes, buffer-drain method, and reward-stamping method. It avoids the GPU-dependent module imports. Run it from the article directory; --fetch downloads the two public source files into a temporary source directory.

The recorded CPU receipt contains eight passing checks, including a blocked producer released by filtering, epoch accounting, checkpoint capacity restoration, the long-group trace, buffer exhaustion, and reward truncation/EOS branches. Those are executed method checks, not a full upstream test-suite pass. A custom chat-template branch returns the final scalar reward instead of the per-token vector, so a generator extension must test its selected path.^{[8]Reference 8SkyRL Gym Generator (955c3e2)https://github.com/NovaSky-AI/SkyRL/blob/955c3e23bf8939b28a9d6c424308a762dcb8a2e7/skyrl/train/generators/skyrl_gym_generator.py}

No CUDA model, Ray cluster, vLLM HTTP server, real SQL environment, weight transfer, or learning curve was executed for this review. The simulated trainer also needs inference endpoints and skips actual optimization and broadcast; it isn't evidence of correct distributed updates. Before a real experiment, run the revision's small synchronous recipe, verify a save/resume round trip, then compare async against that baseline at equal evaluated prompts and GPU-hours.

Strengths and weaknesses

At this point the architecture can be judged by a run, not by its component count. The first table asks what each seam buys; the evidence column names the measurement that can confirm it.

Strengths

The useful sequence is incremental: start with synchronous GRPO, add a custom generator, then test async generation while preserving output fields. Each experiment changes one boundary at a time, so an ablation can point to a seam instead of a pile of infrastructure changes.

Weaknesses and sharp edges

These edges define the admission bar for a serious RL experiment: reproducible environment state, traceable policy versions, bounded queues, and an explicit stop when rewards or sync health become untrustworthy. The operator section turns that bar into evidence you can collect before burning more GPU hours.

Research papers and design lineage

The core path is now complete. Keep the papers as a second reading layer: they explain why a workload matters and what was measured, while the code shows which assumptions became interfaces and which remain configuration constraints.

This table supplies research context rather than a leaderboard. DAPO and AReaL cover related systems, while SkyRL provides a framework for testing environment and execution-plan variants. Compare exact task, base model, reward, sampling, and update settings before importing a result.

Admit a run and recover it

Before launching a serious SkyRL run, write its evidence contract. If one answer is unknown, use a smaller smoke job before spending more GPU hours. The order matters: trust the reward, identify the policy, bound overlap, then rehearse recovery.

Gate 1: make reward traceable

Define one episode and its termination condition. Record which tokens receive policy loss and whether reward is scalar, per-turn, or per-token. Then ask whether the evaluator can give every sample the same score and whether tool outputs are reproducible under a fixed seed and environment snapshot. A zero-variance group or nondeterministic tool can flatten a run before optimization gets a chance to help.

Gate 2: name every weight owner

Write down which process owns training, reference, and sampler weights, whether training and generation are colocated or disaggregated, and which inference and weight-sync backends are selected. After every sync, require every serving replica to report the same policy version. Pin model, tokenizer, CUDA, Ray, and dependency versions so a replay has a meaningful environment.

Gate 3: bound async work

Set mini-batch size, generation-worker count, and max_staleness_steps together. Calculate completed-group plus in-flight headroom as $B(S+1)$, choose the pause mode (keep, abort, or wait), and check whether use_cache_salt isolates prefix-cache blocks across policy versions. Decide which trajectories are filtered as lag rises, how generation failures surface, and which metric stops the run automatically.

Gate 4: keep a recovery packet

A replayable trajectory joins prompt, actions, observations, rewards, and policy version. Logs separate environment latency from inference latency. Require checkpoint saves to be atomic and resumable, and keep a rollback path for reward or sync regressions. Finally, name the offline slice that must pass before online gains count as learning.

🎯 Production tip: Log trajectory ID, scheduled policy step, consumed policy step, reward components, stop reason, and environment revision together. A throughput chart without those joins fails to explain a bad update.

When the extra seams pay off

Choose this stack when the research question includes an environment, a changing rollout policy, or a nontrivial training and inference plan. Tool-use agents, verifier-backed reasoning, Text-to-SQL, terminal tasks, and sync-versus-async experiments all cross those boundaries.

A smaller trainer fits a static supervised dataset with no environment or rollout control. SkyRL's interfaces add coordination cost, so their value starts when one of those boundaries is part of the question.

There is a second fork when policy or data must stay outside your GPU cluster. SkyRL-Tinker offers a service-like API with local hardware control, but its compatibility boundary still needs a pinned integration test.^{[15]Reference 15SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The practical handoff is hybrid: prototype reward and environment logic in a synchronous local run, validate trajectory fields and evaluator slices, then move to disaggregated or fully async execution after a baseline and staleness budget exist. That order lets a later throughput win answer a useful question: did the system learn more, or did it only move more stale tokens?

Review an extension before running it

Evaluation rubric

• Foundational: Trace one response through token IDs, observation masks, reward placement, and the final stop reason without treating a tool observation as a sampled action.
• Intermediate: Reconstruct accepted, running, filtered, and consumed counts, including a long group whose scheduling lag exceeds the aggregate budget.
• Advanced: Separate backend/API compatibility, sampler-version evidence, cache behavior, and held-out learning outcomes when accepting an async change.

Follow-up questions

A two-group mini-batch has one useful group left when the epoch is exhausted. What does sample_full_batch imply for training and resume?

Answer

The incomplete mini-batch is discarded rather than padded into an update. Its prompt IDs are marked consumed so they aren't regenerated on resume. Report discarded and filtered groups separately from trained groups; the nominal epoch step count is only an upper bound.

The client increments its weight-version counter and rewards improve. Has every sampler loaded the new weights?

Answer

No. A local counter records control progress, and reward can improve for unrelated reasons. Check each replica's loaded version and a post-sync probe, then establish the pause boundary for in-flight requests. Unknown sampling weights can't be repaired merely by labeling trajectories off-policy.