Deep Dive - SkyRL

When a model must learn to use a terminal, query a database, or browse a changing website, a text-only training loop is not enough. The model needs an environment that can execute actions, return observations, and score the result. SkyRL is a codebase for wiring that loop without forcing every new environment or inference backend into one giant trainer.

The project is useful to read after a serving deep dive because it makes the opposite boundary visible. A serving engine asks how to answer requests efficiently. SkyRL asks how to create, score, and learn from requests while the model and its sampler keep changing. Coordination dominates: rollout data, policy versions, GPU placement, reward semantics, and failures must agree.

This lesson follows the current unified repository, not one historical release. The repository is being reorganized under skyrl/, while older skyrl-train, skyrl-tx, skyrl-gym, and skyrl-agent packages remain important pieces of the lineage.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

The project in one sentence

SkyRL is a full-stack reinforcement-learning library for large language models (LLMs). Its central design is a set of interfaces around a trajectory generator, environment, inference engine, and trainer, plus a conceptual controller role for placement, initialization, and control flow. Current training control still lives in trainer.py; extracting a standalone controller component remains work in progress. Each boundary carries explicit inputs and outputs instead of making the trainer know every task implementation.^{[2]Reference 2SkyRL Overviewhttps://docs.skyrl.ai/docs/getting-started/overview}

That choice changes what counts as an extension. Adding a Text-to-SQL task should mean implementing an environment or generator, not editing advantage calculation. Switching from a local vLLM worker to a remote OpenAI-compatible endpoint should mean selecting an inference backend, not rewriting reward processing. Moving from one-step pipelining to fully asynchronous training should change scheduling and freshness controls, not the shape of a trajectory.

The project is not a single algorithm. The current stack includes PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and several policy-gradient variants. It supports FSDP (Fully Sharded Data Parallel) and Megatron training backends, vLLM and SGLang inference, custom generators, Gymnasium-style environments, and multiple weight-sync paths.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

💡 Key insight: SkyRL's unit of reuse is the interface seam. If the seam preserves prompt, token, reward, mask, and version semantics, the implementation behind it can change without changing the learning loop.

Project identity and lineage

SkyRL is developed by NovaSky AI at the Berkeley Sky Computing Lab, with collaboration from Anyscale. The repository acknowledges compute support from organizations including Databricks, NVIDIA, Lambda Labs, AMD, AWS, Modal, and Daytona. Treat that list as project provenance, not as a claim that each organization owns or maintains every subsystem.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

The lineage is visible in the repository layout and release notes:

The project also credits ideas and code from veRL, OpenRLHF, Search-R1, OpenReasonerZero, and NeMo-RL. That list maps the ecosystem without making SkyRL a drop-in replacement for any one project.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Start with the mental model

A three-stage conceptual loop captures the architecture: a controller role sets up execution, a Generator coordinates inference and environments to produce trajectories, and a Trainer turns trajectories into gradients. Current control flow remains in trainer.py, while InferenceEngine and Environment stay replaceable boundaries inside the Generator.

The loop is not "model in, loss out." A multi-turn trajectory can include token IDs, tool calls, observations, rewards, per-token masks, and a stop reason. The trainer needs enough metadata to know which tokens should receive learning signal and which tokens belong to an environment transition.

Vocabulary at the seams

SkyRL's current GeneratorInput includes prompts, environment classes, optional environment extras, sampling parameters, trajectory IDs, and batch metadata. GeneratorOutput includes prompt and response token IDs, rewards, loss masks, stop reasons, rollout metrics, optional log probabilities, trajectory IDs, and timing splits. Reading these structures is faster than guessing from a diagram.^{[2]Reference 2SkyRL Overviewhttps://docs.skyrl.ai/docs/getting-started/overview}

The architecture in one request

A short math task and a long tool-use task share this infrastructure. A generator may call an environment many times before returning one output. The trainer still receives one normalized output object.

The feedback edge is deliberate. Environments can return another observation, and fully asynchronous training can buffer a completed group while the trainer is busy. The failure edge is part of the design too: zero-variance groups, stale rollouts, broken tools, or failed weight transfers must be visible rather than silently promoted into a gradient.

Read the main code paths

The current trainer lives in skyrl/train/trainer.py. Its regular loop initializes weight-sync state, saves weights for the sampler, optionally evaluates, generates a batch, post-processes rewards, converts outputs to training input, computes log probabilities, advantages, and returns, then trains policy and, when configured, critic workers. It also logs trajectories and timing around each stage.

The generator contract lives in skyrl/train/generators/base.py. The abstract method is asynchronous, which matters even in a synchronous RL schedule: an environment may await a browser, a remote verifier, or a tool process without blocking every other Python task. A custom generator has one required method, generate(), and returns output in the same order as the input batch.

The environment path is intentionally outside the trainer. SkyRLGymGenerator adapts Gymnasium-style tasks, but examples also integrate verifiers, OpenEnv, Harbor, and Terminal-Bench. The agent layer adds another adapter for long-horizon loops. This keeps a small core with many task-specific edges instead of a trainer fork per benchmark.

A concrete trajectory

Consider a Text-to-SQL prompt: "List active customers with more than three unresolved incidents." A generator can ask the model for a SQL action, pass the action to a sandboxed database environment, return the database error or rows as an observation, and let the model repair its query. The reward can be computed from execution correctness and answer shape, while the loss mask selects model-generated tokens.

For one sampled trajectory, a conceptual output might look like this:

The IDs and masks are not decoration. A multi-turn batch can flatten several conversations, while step-wise training can return one turn at a time and mark whether the episode is complete. Losing that metadata creates silent credit-assignment bugs.

The trainer's order of operations

SkyRL's regular loop follows this ordering:

1. Prepare prompts and sampling parameters.
2. Generate trajectories through the configured generator and inference engine.
3. Apply dynamic sampling rules, including retry or filter behavior.
4. Post-process rewards and merge step-wise turns when enabled.
5. Convert GeneratorOutput into training input.
6. Compute current log probabilities, values when a critic is configured, rewards, advantages, and returns.
7. Run policy updates and optional critic updates.
8. Save checkpoints, evaluate, and synchronize updated weights for the next sampling window.

The order protects an important invariant: the trainer computes learning quantities from a known representation of the sampled trajectory. Environments finish their mutations before optimization starts.

Synchronous RL: the reference schedule

Synchronous reinforcement learning is a useful baseline because it makes policy freshness easy to explain. Generate a batch with policy version $\pi_k$, wait until all responses and rewards arrive, update the policy to $\pi_{k+1}$, then send new weights to the sampler. The next batch starts only after the sync boundary.

The schedule has a clean policy-gradient interpretation. For a response $y$ sampled from prompt $x$, the trainer uses an objective of the form:

$$L(\theta) = -\mathbb{E}_{(x,y) \sim \pi_k}\left[A(x,y) \log \pi_\theta(y \mid x)\right],$$

where $A(x,y)$ is an advantage estimate. In plain language, a response with positive advantage receives more probability, while a response below the baseline receives less. PPO adds a clipped ratio to prevent one update from moving too far from the sampling policy.

GRPO replaces a learned value model with relative scores within a group of samples for the same prompt. That makes the environment reward and group construction central. A group with identical rewards has no relative signal, which is why SkyRL's dynamic and zero-variance filters matter.

What synchronous training buys

The cost is visible when tools are slow or trajectories have uneven lengths. One browser task can hold an entire step open while other GPUs wait. Synchronous mode remains the right first implementation for a new environment because it limits the number of moving parts.

One-step async and fully async

SkyRL exposes more than one meaning of "async." One-step off-policy pipelining overlaps generation for a future batch with optimization of the current batch. Fully asynchronous training goes further: several generation workers keep producing groups, a bounded buffer holds completed groups, and the trainer consumes mini-batches while generation continues on another scheduling path.^{[6]Reference 6Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

This distinction matters when reading benchmarks or debugging a configuration. One-step overlap still has a relatively obvious handoff. Fully async requires a freshness policy, a queue-capacity policy, pause and resume around weight sync, and metrics that tell you how old each group was when consumed.

The fully async implementation creates generation tasks per epoch and places completed groups in an asyncio.Queue. Its buffer is bounded using the mini-batch size and a max_staleness_steps budget. A staleness manager limits producer capacity, records submitted, running, accepted, and filtered groups, and validates state at epoch end. That is a control system, not a free speed switch.^{[6]Reference 6Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

Staleness as a budget

Let $s_i$ be the trainer's current global step when group $i$ is consumed and let $g_i$ be the step when generation was scheduled. SkyRL records:

$$\operatorname{staleness}_i = s_i - g_i.$$

If max_staleness_steps = 2, SkyRL uses that budget to size producer capacity and bound aggregate backlog. The value doesn't impose a hard per-group rejection threshold: an individual group can still arrive with staleness 3. Track the full distribution plus submitted, running, accepted, and filtered counts. Frequent high-lag groups mean the producer is outrunning the consumer or generation latency is too uneven.^{[6]Reference 6Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

The queue capacity follows the same reasoning. With mini-batch size $B$ and staleness budget $S$, the completed generation buffer is bounded around $B(S+1)$. A larger buffer can keep GPUs busy but also increases the opportunity for old policy data. A smaller buffer limits freshness risk but makes the trainer wait more often.

The fully async path has explicit constraints. Batched generate() calls are not supported because pause and continue need a single in-flight generation boundary. Callback support is not wired into the fully async trainer in the current code. The regular dynamic_sampling mode and colocating all training and inference workers are not supported. Fully async instead offers sample_full_batch with zero-variance filtering for its native filtering path. These are real tradeoffs, not documentation footnotes.

AReaL and DAPO as research context

SkyRL's fully async design is close to a broader research question: can language-model RL keep GPUs busy without letting policy lag destroy the training signal? AReaL, a large-scale asynchronous RL system for language reasoning, provides a useful vocabulary for in-flight generation and staleness control.^{[7]Reference 7AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoninghttps://arxiv.org/abs/2505.24298}

DAPO is another important reference point. Its open system describes scalable RL ingredients such as dynamic sampling and token-level policy-gradient choices. SkyRL exposes related knobs, but an implementation flag is not proof that two systems produce identical experiments. Compare data filtering, reward normalization, rollout policy version, and evaluation protocol before claiming parity.^{[8]Reference 8DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

Treat papers as design evidence, then trace the local code that realizes each assumption. A queue can overlap work, but only metrics can show whether the resulting batches are fresh enough and diverse enough to learn.

Inference and weight synchronization

SkyRL separates training and sampling because they have different hardware and runtime needs. The training backend may be FSDP or Megatron, while the inference backend may be vLLM, SGLang, or a custom OpenAI-compatible data plane paired with control-plane weight-sync endpoints. Training and generation can be colocated on one set of GPUs or disaggregated across workers.^{[9]Reference 9SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

The official inference architecture separates a control plane from a data plane. Generation requests travel through the HTTP data plane, while routing, placement, health, and weight operations belong to the control plane. This split lets the trainer ask a remote engine to pause, load, or resume without putting model internals into every generator.

Weight synchronization is the boundary where "on-policy" becomes an operational claim. SkyRL's project lineage and legacy docs describe NCCL (NVIDIA Collective Communications Library), Gloo-backed transfer, and checkpoint-and-load paths. The current unified path defaults to NCCL, so verify the backend implemented at the exact revision you deploy. NCCL can be fast on compatible GPU topology, while checkpoint-based deployment is easier to inspect across process boundaries but costs latency and storage. No path removes the need to record policy versions.

A sync failure path

Suppose the trainer finishes step 12, but one inference replica still serves step 11 weights. The next group can contain a mixed policy version depending on routing. A reliable operator response is:

1. Pause new generation if the endpoint supports it.
2. Record trainer step and sampler version for every replica.
3. Check transfer logs and replica health.
4. Retry or roll back the sync operation.
5. Resume only when all replicas report the expected version.

If the system must continue during a degraded replica, mark its trajectories as off-policy and route them through an explicit correction or discard policy. A successful request alone doesn't make them on-policy.

The same boundary affects key-value (KV) cache state. Inference engines may need to clear or preserve cache entries when weights change. SkyRL exposes configuration for clearing the KV cache on sync and for offloading KV state during a transfer. The correct setting depends on serving backend behavior and whether cached activations remain valid after an update.

Tinker: a stable API over changing hardware

Tinker is a training API that presents model training and sampling as a service-like interface. SkyRL tx implements a Tinker-compatible backend so users can run a Tinker-style program on their own hardware. The current unified package carries that work under skyrl while the old skyrl-tx directory documents the migration.^{[10]Reference 10SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The Tinker architecture is easiest to understand as a stack:

The value is not that all hardware looks identical. The value is that a recipe can remain stable while placement and backend decisions change below it. That is useful for a team comparing a local workstation, a Ray cluster, and a managed service, as long as the API contract exposes the important differences.

Tinker also clarifies a useful split between control and data. A user submits an operation and receives a future-like handle. The engine schedules the work, returns results, and persists checkpoints. For agent training, generation can remain an explicit operation rather than an unbounded callback hidden inside the optimizer.

The weakness is contract drift. If a Tinker recipe assumes a method or shape not implemented by the selected backend, the abstraction fails at runtime. Pin the SkyRL commit, backend versions, model tokenizer, and recipe configuration for a reproducible experiment. Read the generated API docs and the backend implementation when a behavior matters.

SkyRL-Agent: long-horizon work belongs above the trainer

Short benchmark prompts are not enough for SWE (software-engineering) agents, web researchers, or terminal users. These agents make many tool calls, carry state across turns, and can fail for reasons outside the language model. SkyRL-Agent adds a reusable layer for tasks, tools, dispatch strategies, and training backends. It can connect to OpenAI-compatible serving such as vLLM, veRL, SkyRL-Train, or Tinker with a configuration change.^{[3]Reference 3SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

The agent layer keeps environment-specific logic close to the task. Browser tools return page observations, code-execution tools return stdout and process status, and finish tools mark episodes complete. The dispatcher can run work asynchronously, while the training backend consumes a normalized trajectory.

The paper frames the problem as efficient training for multi-turn, long-horizon language agents. Its systems contribution brings tool calls, replayable histories, asynchronous dispatch, and backend selection into one training and evaluation interface.^{[3]Reference 3SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agenthttps://arxiv.org/abs/2511.16108}

Application map

The same library can support all six, but the reward contract is not interchangeable. SQL execution may yield an exact reward; research agents may combine evidence coverage, citation validity, and answer quality; terminal tasks may depend on flaky or expensive tests. Keep those semantics documented next to the environment.

Four coordinated mechanisms

1. Typed seams keep research moving

Researchers often need to change the environment, the sampling strategy, and the optimizer in the same week. A stable GeneratorInput and GeneratorOutput make those changes composable. The trainer can consume a new generator without learning the generator's internal tool protocol.

The seam also makes review concrete. A pull request can answer: Did every response have a mask? Did every trajectory receive an ID? Are rewards scalar or per-turn? Did the generator preserve input order? These are stronger questions than "does the agent seem better?"

2. Environment grounding turns text into feedback

Self-grading SQL queries or code patches provide weak evidence. SkyRL puts an executable environment in the loop: it observes an action, performs a bounded operation, and returns a result that can be logged and scored.

Grounding alone leaves reward failure modes. Permissive SQL databases can leak answers, terminal sandboxes can expose network state, and browser evaluators can over-reward superficial page matching. The environment contract moves correctness into code, where it can be tested and audited.

3. Async overlap attacks the idle-time problem

Long-horizon trajectories are uneven. Fully async generation lets short episodes continue while a long episode is still waiting for a tool. The trainer consumes ready groups from a bounded buffer. This can improve utilization without making the queue infinite.

The improvement is conditional. If policy updates are fast and tools are slow, overlap can help. If the queue grows beyond the freshness budget, the same overlap becomes off-policy drift. SkyRL makes that tradeoff explicit with staleness metrics and capacity controls.^{[6]Reference 6Fully Asynchronous Traininghttps://docs.skyrl.ai/docs/tutorials/fully_async}

4. Weight sync is a first-class operation

Training and inference have different memory layouts, process boundaries, and failure modes. Treating weight sync as a named operation makes it possible to pause generation, transfer parameters, verify versions, and resume. That is more honest than assuming model.load_state_dict() works across every backend.

This design also enables disaggregation. A generation cluster can serve requests while a training cluster performs updates, provided the control plane can coordinate versions and health. The architecture documentation calls out routing, placement, and sync as separate concerns for this reason.^{[9]Reference 9SkyRL Inference Architecturehttps://docs.skyrl.ai/docs/getting-started/inference_architecture}

A worked policy update

Use a tiny group to see what the trainer needs. Prompt two models with the same question: "What is 2 + 2?" Suppose the environment returns reward 1 for 4 and reward 0 for 5. Let the group mean be $0.5$. Their relative advantages are $+0.5$ and $-0.5$.

The update pushes probability toward the first response and away from the second. A simplified group-relative loss is:

$$L = -\frac{1}{2}\left[(+0.5)\log \pi_\theta(4\mid x) + (-0.5)\log \pi_\theta(5\mid x)\right].$$

That equation hides three implementation details:

1. The group must contain comparable samples for the same prompt.
2. The reward must be aligned with the environment's true success condition.
3. The loss mask must identify the response tokens that policy optimization should change.

If both responses receive reward 1, the relative advantages are zero. Additional samples won't repair an evaluator that assigns identical outcomes. Filter or redesign the task before tuning learning rate.

This configuration sketch exposes knobs rather than presenting a complete runnable experiment. It uses the conceptual names from SkyRL's configuration model.

Read this file as a contract review checklist. Confirm that n_samples_per_prompt creates a meaningful group, that zero-variance filtering is compatible with the selected async mode, and that the inference endpoint supports the requested sync backend. Then consult the repository's recipe for model, tokenizer, Ray, CUDA, and dataset requirements.^{[1]Reference 1SkyRLhttps://github.com/NovaSky-AI/SkyRL}

Strengths and weaknesses

Strengths

The strongest advantage is change isolation. A team can start with synchronous GRPO, then add a custom generator, then test async generation while preserving the same output fields. That makes ablation work cheaper and code review more focused.

Weaknesses and sharp edges

These are not reasons to avoid SkyRL. They define what "production ready" means for an RL experiment: reproducible environment state, traceable policy versions, bounded queues, and a way to stop when rewards or sync health become untrustworthy.

Research papers and design lineage

Read the local repository and its papers together. Papers explain why a workload matters and what was measured; code shows which assumptions became interfaces and which remain configuration constraints.

This table supplies research context rather than a leaderboard. DAPO and AReaL cover related systems, while SkyRL provides a framework for testing environment and execution-plan variants. Compare exact task, base model, reward, sampling, and update settings before importing a result.

Operator checklist

Before launching a serious SkyRL run, write down the answers below. If one answer is unknown, run a smaller smoke job before spending more GPU hours.

Data and reward

• What is one episode, and what ends it?
• Which tokens receive policy loss, and why?
• Is reward scalar, per turn, or per token?
• Can the evaluator produce identical rewards for every sample?
• Are tool outputs reproducible under a fixed seed and environment snapshot?

Runtime and placement

• Which process owns training weights, reference weights, and sampler weights?
• Are training and generation colocated or disaggregated?
• Which inference backend and weight-sync backend are selected?
• Does every serving replica report the same policy version after sync?
• Are model, tokenizer, CUDA, Ray, and dependency versions pinned?

Async controls

• What are mini-batch size, generation-worker count, and max_staleness_steps?
• What is the maximum completed-group buffer size?
• Which events pause generation during sync?
• Which trajectories are filtered or accepted as staleness rises, and how are generation failures surfaced?
• What metric triggers an automatic stop?

Evidence and recovery

• Can you replay one trajectory from prompt, actions, observations, rewards, and policy version?
• Do logs separate environment latency from inference latency?
• Is checkpoint save atomic, and can a new process resume it?
• What is the rollback path when reward or sync health regresses?
• Which offline slice must pass before trusting online gains?

🎯 Production tip: Log trajectory ID, scheduled policy step, consumed policy step, reward components, stop reason, and environment revision together. A throughput chart without those joins fails to explain a bad update.

When to choose SkyRL

Choose SkyRL when the research question includes an environment, a changing rollout policy, or a nontrivial training and inference execution plan. Strong fits include tool-use agents, reasoning tasks with verifiers, Text-to-SQL, terminal tasks, and experiments that compare synchronous with asynchronous RL.

Choose a smaller trainer when the task is a static supervised dataset and no environment or rollout control is required. The modular interfaces add coordination cost; they pay off when you need to change one of those boundaries.

Choose a managed service when operational simplicity matters most and policy or data must stay outside your own GPU cluster. SkyRL-Tinker offers a service-like API with local hardware control, but its compatibility boundary still needs a pinned integration test.^{[10]Reference 10SkyRL-Tinker Architecturehttps://docs.skyrl.ai/docs/tinker/architecture}

The mature decision is often hybrid: prototype reward and environment logic in a synchronous local run, validate trajectory fields and evaluator slices, then move to disaggregated or fully async execution only after you have a baseline and a staleness budget.

Key takeaways

• SkyRL is a full-stack LLM RL library organized around replaceable interfaces.
• GeneratorInput and GeneratorOutput make prompts, trajectories, rewards, masks, IDs, and timing explicit.
• Environment grounding moves reward computation into executable, testable code.
• Synchronous RL is the clearest baseline for a new task.
• Fully async RL overlaps generation and training, then pays for that overlap with queue and staleness controls.
• Weight synchronization is a policy-version boundary, not an implementation detail.
• SkyRL-Agent handles long-horizon tools above the core trainer, while Tinker provides a service-shaped API over changing backends.
• DAPO and AReaL offer useful research context, but experiments must be compared under matching protocols.

Before you call a run successful, inspect reward diversity, policy-version joins, staleness percentiles, environment latency, checkpoint resume, and a frozen evaluation slice. These checks tell you whether the system learned or whether the infrastructure merely moved more tokens.