Reinforcement Learning Basics

Predicting whether an access-review request needs human review is a one-step decision. A real access assistant must choose a sequence: approve now, request evidence, or route to a reviewer; then react when evidence arrives or the user abandons the request.

Reinforcement learning (RL) studies that sequential decision problem. The agent chooses actions, the environment produces next states and rewards, and the objective is high total reward over an episode. One small access-review workflow gives every equation a concrete operational meaning.^{[1]Reference 1Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

You need Python, small NumPy arrays, and weighted averages. The rewards below are teaching scores, not a production business objective.

From Prediction to an Episode

Access request acct_10234 is stale and policy P-7 requires review unless strong evidence arrives. Three initial actions are available:

An episode starts at new_request and ends at closed. Suppose evidence definitely arrives and rewards after the first action are:

RL compares routes with the discounted return

$$G_0 = r_0 + \gamma r_1 + \gamma^2 r_2 + \cdots$$

where $0 \leq \gamma \leq 1$. Values below $1$ discount delayed reward; we use $\gamma = 0.90$.

With certain evidence arrival, request_evidence earns $-4 + 0.90(70) = 59$, just above human_review at $54$. The route decision depends on future consequences as well as immediate cost.

Define the MDP

A Markov Decision Process (MDP) supplies the pieces needed to score a policy:

• States $S$: what is known at decision time.
• Actions $A(s)$: valid decisions in a state.
• Transitions $P(s' \mid s, a)$: probabilities of the next state.
• Rewards $R(s, a, s')$: numeric consequence attached to a transition.
• Discount $\gamma$: how strongly delayed consequences count.

Our first model is deliberately deterministic:

The state must contain information needed for the next decision. new_request and evidence_ready are different states because approving after evidence isn't the same decision as approving without it. Under the Markov assumption, once the current state and action are known, earlier history doesn't change the transition distribution.^{[1]Reference 1Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

Value and Bellman Backups

A policy $\pi(a \mid s)$ describes how an agent chooses actions. The optimal value $V^*(s)$ is the best expected future return achievable from state $s$.

Start at the end:

• closed has no future reward, so $V^*(\texttt{closed}) = 0$.
• review_queue has one useful action, so V^*(\texttt{review_queue}) = 80.
• evidence_ready chooses between approving for $70$ and reviewing for $-10 + 0.90(80) = 62$, so its value is $70$.
• new_request chooses among $-120$, $-4 + 0.90(70) = 59$, and $-18 + 0.90(80) = 54$, so its value is $59$.

This backward computation is a Bellman backup:

$$V^*(s) = \max_{a \in A(s)} \sum_{s'} P(s' \mid s,a) \left[R(s,a,s') + \gamma V^*(s')\right]$$

Value iteration: plan when the model is known

Hand backups work because this MDP is tiny. Value iteration applies the same backup repeatedly to every non-terminal state until values stop changing. It assumes transition probabilities and rewards are already specified.

The first sweep learns terminal-adjacent outcomes. The second propagates them back to the initial request. Value methods can assign credit to an early evidence request whose payoff arrives later.

Transition Risk Can Reverse a Decision

The deterministic model hid an operational risk: some users never provide the requested evidence. Suppose request_evidence has these transitions:

The expected value of requesting evidence is now

$$0.75[-4 + 0.90(70)] + 0.25[-54] = 30.75$$

while direct review is still $-18 + 0.90(80) = 54$. Modeling abandonment flips the recommended action.

Expected values describe many similar requests, not a guaranteed outcome for one user. Simulation makes that variation visible.

Production probabilities can't be invented from one example. They must be estimated from logged episodes, monitored as policy changes alter user behavior, and evaluated on future data.

Q-learning: learn from experience

Value iteration required a transition table. If the router instead observes episodes, it can learn action values $Q(s,a)$ from sampled transitions:

$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'}Q(s',a') - Q(s,a) \right]$$

This is the tabular Q-learning update. Under its standard finite-setting coverage and learning-rate conditions, it converges to optimal action values; practical systems still face limited data, drifting behavior, and unsafe exploration.^{[2]Reference 2Q-learninghttps://doi.org/10.1007/BF00992698}

The lab below uses epsilon-greedy coverage and a visit-count learning rate that shrinks as evidence accumulates.

The sampled evidence-route estimate lands near the model-based value 30.75, not exactly on it, because this finite run still contains transition noise. The declining learning rate gives later samples less power to overwrite accumulated evidence.

Exploration matters. If an agent always selects its current best action, an unlucky first impression can prevent it from finding the genuinely stronger route. The next mini-lab collapses the completed routes into one-state action returns to isolate that point.

Unsafe actions can't be explored casually in a live approval system. Offline logs, conservative policy constraints, human approval, or simulation are needed before a learned policy controls costly decisions.

Reward Specification Is Part of the System

An RL policy optimizes the reward it receives, not the intent in a system design document. A flawed score that rewards only short handling time can prefer unsupported automatic approvals.

Useful reward design records user outcome, policy violations, delays, reviewer cost, abuse risk, and uncertainty. It also audits rare high-harm failures separately from average reward. A high mean return doesn't excuse repeated unsupported approvals.

Policy Gradients and the LLM Bridge

Tabular Q-learning stores a number for every state-action pair. One option for large state or action spaces is a parameterized policy $\pi_\theta(a \mid s)$. REINFORCE increases the probability of sampled actions whose returns exceed a baseline:

$$\nabla_\theta J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N} \nabla_\theta \log \pi_\theta(a_i \mid s_i) \left(G_i - b\right)$$

An action-independent baseline can lower variance without changing the expected policy gradient.^{[3]Reference 3Simple statistical gradient-following algorithms for connectionist reinforcement learninghttps://link.springer.com/article/10.1007/BF00992696}

To expose the arithmetic without a neural network, the next batch includes one completed route for each action and uses their mean return as the baseline. It reuses softmax from classification to turn one logit per action into action probabilities. The policy gradient then shifts probability toward actions with positive advantage and away from actions with negative advantage. Real training estimates that gradient from sampled trajectories.

For a language model, the state can include a prompt and generated prefix, while actions are token choices. In InstructGPT, human comparisons trained a reward model and PPO optimized a policy against that learned reward with additional controls; that's a much larger system, but the underlying sequence-and-reward vocabulary begins here.^{[4]Reference 4Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Evaluate a Policy on Held-Out Episodes

Training reward isn't a deployment guarantee. A routing policy should be compared on later requests whose outcomes didn't influence its rules, rewards, or thresholds. Use a train, validation, and test split: fit on earlier episodes, tune on separate episodes, then reserve a final untouched period. The next chapter treats that evaluation design carefully; this small table shows the question to ask.

This table is illustrative, not evidence that a production policy works. Real evaluation must prevent future outcomes, repeat users, and policy-tuning decisions from leaking into the held-out estimate.

Practice tasks

1. Change abandonment probability in stochastic-transition-backup.py. At what probability does human_review overtake request_evidence?
2. Add a deny_request action and choose rewards that discourage both unsupported approvals and unfair denials. Explain every term.
3. Modify the Q-learning lab so high-risk actions require a reviewer approval flag. Compare learned reward with and without that constraint.
4. Replace the hand-authored held-out episodes with a time-ordered sample. List features that wouldn't exist at decision time.

Practice guidance

1. If abandonment probability is p, evidence-request value is (1 - p)(59) + p(-54) = 59 - 113p. Human review overtakes it when p > 5/113, about 4.4%.
2. A defensible deny_request reward should include policy correctness and user harm, not processing speed alone. Check whether any shortcut action wins for the wrong reason.
3. An approval constraint should block or reroute an unsafe action before the environment step. Compare mean reward and count of blocked unsafe attempts; don't optimize mean alone.
4. Likely leakage fields include evidence_received_at, review outcome, approval reversal, and events caused by router action. Keep the decision-time snapshot separate from the later outcome log.

Key concepts

• episode and discounted return
• Markov Decision Process
• Bellman backup and optimal value
• value iteration with a known model
• stochastic transitions and expected value
• Q-learning from sampled transitions
• exploration and safety constraints
• reward specification
• policy gradients
• held-out policy evaluation

Evaluation rubric

• Foundational: Computes discounted returns and explains why an evidence request can beat an immediate approval.
• Intermediate: Defines the access-routing MDP, implements Bellman backups and value iteration, and explains why abandonment risk changes the policy.
• Advanced: Implements tabular learning, critiques reward design and exploration safety, and connects policy-gradient updates to LLM post-training without overstating the analogy.

Follow-up questions

Common pitfalls

• Treating a classifier probability as a complete policy when actions change later observations and outcomes.
• Rewarding speed while omitting policy violations, user harm, or review cost.
• Planning from invented transition probabilities instead of measured outcomes.
• Exploring unsafe routes on live users without approval or conservative constraints.
• Requesting policy quality from training episodes or from a holdout already used to tune rewards.
• Saying RLHF is identical to the toy MDP rather than a scaled system with learned rewards and optimization controls.