Reinforcement Learning Basics

Predicting whether an access-review request needs human review is a one-step decision. A real access assistant must choose a sequence: approve now, request evidence, or route to a reviewer; then react when evidence arrives or the user abandons the request.

Reinforcement learning (RL) studies that sequential decision problem. The agent chooses actions, the environment produces next states and rewards, and the objective is high total reward over an episode. One small access-review workflow gives every equation a concrete operational meaning.^{[1]Reference 1Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

You need Python, small NumPy arrays, and weighted averages. The rewards below are teaching scores, not a production business objective.

From prediction to an episode

Access request acct_10234 is stale and policy P-7 requires review unless strong evidence arrives. Three initial actions are available:

An episode starts at new_request and ends at closed. Suppose evidence definitely arrives and rewards after the first action are:

RL compares routes with the discounted return

$$G_0 = r_0 + \gamma r_1 + \gamma^2 r_2 + \cdots$$

where $0 \leq \gamma \leq 1$. Values below $1$ discount delayed reward; we use $\gamma = 0.90$.

With certain evidence arrival, request_evidence earns $-4 + 0.90(70) = 59$, just above human_review at $54$. The route decision depends on future consequences as well as immediate cost.

Define the MDP

A Markov Decision Process (MDP) supplies the pieces needed to score a policy:

• States $S$: what is known at decision time.
• Actions $A(s)$: valid decisions in a state.
• Transitions $P(s' \mid s, a)$: probabilities of the next state.
• Rewards $R(s, a, s')$: numeric consequence attached to a transition.
• Discount $\gamma$: how strongly delayed consequences count.

Our first model is deliberately deterministic:

The state must contain information needed for the next decision. new_request and evidence_ready are different states because approving after evidence isn't the same decision as approving without it. Under the Markov assumption, once the current state and action are known, earlier history doesn't change the transition distribution.^{[1]Reference 1Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

Why are new_request and evidence_ready separate states?

Answer

The evidence changes which actions are safe and worth taking. In new_request, auto-approving violates the review requirement; in evidence_ready, the same action closes a supported request.

Value and Bellman backups

A policy $\pi(a \mid s)$ describes how an agent chooses actions. The optimal value $V^*(s)$ is the best expected future return achievable from state $s$.

Start at the end:

• closed has no future reward, so $V^*(\texttt{closed}) = 0$.
• review_queue has one useful action, so V^*(\texttt{review_queue}) = 80.
• evidence_ready chooses between approving for $70$ and reviewing for $-10 + 0.90(80) = 62$, so its value is $70$.
• new_request chooses among $-120$, $-4 + 0.90(70) = 59$, and $-18 + 0.90(80) = 54$, so its value is $59$.

This backward computation is a Bellman backup:

$$V^*(s) = \max_{a \in A(s)} \sum_{s'} P(s' \mid s,a) \left[R(s,a,s') + \gamma V^*(s')\right]$$

Why can a Bellman backup reject the action with the largest immediate reward?

Answer

It scores immediate reward plus discounted future value. A tempting action can lose once its probability of entering a costly future state is included.

Value iteration: plan when the model is known

Hand backups work because this MDP is tiny. Value iteration applies the same backup repeatedly to every non-terminal state until values stop changing. It assumes transition probabilities and rewards are already specified.

The first sweep learns terminal-adjacent outcomes. The second propagates them back to the initial request. Value methods can assign credit to an early evidence request whose payoff arrives later.

Transition risk can reverse a decision

The deterministic model hid an operational risk: some users never provide the requested evidence. Suppose request_evidence has these transitions:

The expected value of requesting evidence is now

$$0.75[-4 + 0.90(70)] + 0.25[-54] = 30.75$$

while direct review is still $-18 + 0.90(80) = 54$. Modeling abandonment flips the recommended action.

Expected values describe many similar requests, not a guaranteed outcome for one user. Simulation makes that variation visible.

Production probabilities can't be invented from one example. They must be estimated from logged episodes, monitored as policy changes alter user behavior, and evaluated on future data.

Q-learning: learn from experience

Value iteration required a transition table. If the router instead observes episodes, it can learn action values $Q(s,a)$ from sampled transitions:

$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'}Q(s',a') - Q(s,a) \right]$$

This is the tabular Q-learning update. Under its standard finite-setting coverage and learning-rate conditions, it converges to optimal action values; practical systems still face limited data, drifting behavior, and unsafe exploration.^{[2]Reference 2Q-learninghttps://doi.org/10.1007/BF00992698}

Q-learning is off-policy: the target uses $\max_{a'} Q(s', a')$ (the best action under the current estimate), while the behavior policy that collects data can be $\varepsilon$-greedy or any other exploratory rule. That separation is the contrast with on-policy methods such as SARSA, which bootstrap from the action the behavior policy actually takes next. Staff interview conversations often start with that one sentence.

The lab below uses epsilon-greedy coverage and a visit-count learning rate that shrinks as evidence accumulates.

The sampled evidence-route estimate lands near the model-based value 30.75, not exactly on it, because this finite run still contains transition noise. The declining learning rate gives later samples less power to overwrite accumulated evidence.

Exploration matters. If an agent always selects its current best action, an unlucky first impression can prevent it from finding the genuinely stronger route. The next mini-lab collapses the completed routes into one-state action returns to isolate that point.

Unsafe actions can't be explored casually in a live approval system. Offline logs, conservative policy constraints, human approval, or simulation are needed before a learned policy controls costly decisions.

In tabular Q-learning, why does the update use the largest next-state Q-value rather than only the next action the behavior policy happened to take?

Answer

Q-learning is off-policy: it learns toward the greedy target while the behavior policy may still explore. That separation lets exploratory experience improve the estimated best policy.

Reward specification is part of the system

An RL policy optimizes the reward it receives, not the intent in a system design document. A flawed score that rewards only short handling time can prefer unsupported automatic approvals.

Useful reward design records user outcome, policy violations, delays, reviewer cost, abuse risk, and uncertainty. It also audits rare high-harm failures separately from average reward. A high mean return doesn't excuse repeated unsupported approvals.

Policy gradients and the LLM bridge

Tabular Q-learning stores a number for every state-action pair. One option for large state or action spaces is a parameterized policy $\pi_\theta(a \mid s)$. REINFORCE, a Monte Carlo policy-gradient estimator, increases the probability of sampled actions whose returns exceed a baseline:

$$\nabla_\theta J(\theta) \approx \frac{1}{N}\sum_{i=1}^{N} \nabla_\theta \log \pi_\theta(a_i \mid s_i) \left(G_i - b\right)$$

An action-independent baseline can lower variance without changing the expected policy gradient.^{[3]Reference 3Simple statistical gradient-following algorithms for connectionist reinforcement learninghttps://link.springer.com/article/10.1007/BF00992696}

To expose the arithmetic without a neural network, the next batch includes one completed route for each action and uses their mean return as the baseline. It reuses softmax from classification to turn one logit per action into action probabilities. The policy gradient then shifts probability toward actions with positive advantage and away from actions with negative advantage. Real training estimates that gradient from sampled trajectories.

For a language model, the state can include a prompt and generated prefix, while actions are token choices. In InstructGPT, human comparisons trained a reward model and Proximal Policy Optimization (PPO) optimized a policy against that learned reward with additional controls; that's a much larger system, but the underlying sequence-and-reward vocabulary begins here.^{[4]Reference 4Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Evaluate a policy on held-out episodes

Training reward isn't a deployment guarantee. A routing policy should be compared on later requests whose outcomes didn't influence its rules, rewards, or thresholds. Use a train, validation, and test split: fit on earlier episodes, tune on separate episodes, then reserve a final untouched period. The next chapter treats that evaluation design carefully; this small table shows the question to ask.

This table is an oracle, not ordinary production data. Each episode dict lists rewards for every action, including ones the production router never took. The counterfactual scorecard assumes you already know what would have happened under alternate actions. Real logs usually record only the chosen action's outcome. The next lesson (validation and leakage) makes that limit the centerpiece; production off-policy evaluation needs logged propensities, careful exploration, a validated simulator, or a staged online experiment.

This table is an oracle illustration, not evidence that a production policy works. Real evaluation must prevent future outcomes, repeat users, and policy-tuning decisions from leaking into the held-out estimate, and must not invent rewards for actions the log never took without an explicit counterfactual method.

Practice tasks

1. Change abandonment probability in stochastic-transition-backup.py. At what probability does human_review overtake request_evidence?
2. Add a deny_request action and choose rewards that discourage both unsupported approvals and unfair denials. Explain every term.
3. Modify the Q-learning lab so high-risk actions require a reviewer approval flag. Compare learned reward with and without that constraint.
4. Replace the hand-authored held-out episodes with a time-ordered sample. List features that wouldn't exist at decision time.

Practice guidance

1. If abandonment probability is p, evidence-request value is (1 - p)(59) + p(-54) = 59 - 113p. Human review overtakes it when p > 5/113, about 4.4%.
2. A defensible deny_request reward should include policy correctness and user harm, not processing speed alone. Check whether any shortcut action wins for the wrong reason.
3. An approval constraint should block or reroute an unsafe action before the environment step. Compare mean reward and count of blocked unsafe attempts; don't optimize mean alone.
4. Likely leakage fields include evidence_received_at, review outcome, approval reversal, and events caused by router action. Keep the decision-time snapshot separate from the later outcome log.