RL Bible · Appendix A

Notation & Conventions

Every symbol used in this book, in one place.

This book uses one consistent notation from the first bandit to the last vision-language-action model. When a chapter introduces a symbol, it uses the convention here; when a paper we teach uses different notation, we translate it into this one and say so. Capital letters denote random variables, lowercase letters denote particular values, and bold is reserved for vectors and matrices where the distinction matters.

1. The Agent–Environment Interface

Symbol	Meaning
$t$	discrete time step, $t = 0, 1, 2, \dots$
$\mathcal{S}$	set of states
$\mathcal{A}$ , $\mathcal{A}(s)$	set of actions (available in state $s$ )
$\mathcal{R} \subset \R$	set of rewards
$S_t$	state at time $t$ (a random variable)
$A_t$	action at time $t$
$R_{t+1}$	reward received after taking $A_t$ in $S_t$
$O_t$	observation at time $t$ (partially observable settings)
$\tau$	a trajectory $(S_0, A_0, R_1, S_1, A_1, R_2, \dots)$
$T$	final time step of an episode

We follow Sutton & Barto's convention that the reward for the transition $(S_t, A_t) \to S_{t+1}$ is indexed $R_{t+1}$ , not $R_t$ : the reward arrives with the next state, one tick after the action that earned it. Some papers (and most deep-RL codebases) write $r_t$ for the same quantity; when we quote such a paper we align its indices with ours.

2. Dynamics and the MDP

Symbol	Meaning
$p(s', r \mid s, a)$	joint dynamics: probability of next state $s'$ and reward $r$ given $(s,a)$
$P(s' \mid s, a)$	state-transition probability, $\sum_r p(s', r \mid s, a)$
$r(s, a)$	expected immediate reward, $\E[R_{t+1} \mid S_t = s, A_t = a]$
$r(s, a, s')$	expected reward given the successor state too
$\mu_0(s)$ or $\rho_0(s)$	initial-state distribution
$\gamma \in [0, 1]$	discount factor
$d^\pi(s)$	(discounted) state visitation distribution under $\pi$

An MDP is the tuple $(\mathcal{S}, \mathcal{A}, p, \gamma)$ , or $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$ when transition and reward are given separately. Chapter 3 constructs all of this carefully.

3. Returns, Policies, and Value Functions

Symbol	Meaning
$G_t$	return from time $t$ : $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$
$G_{t:t+n}$	$n$ -step return (Chapter 7)
$G_t^\lambda$	λ-return (Chapter 7)
$\pi(a \mid s)$	stochastic policy: probability of action $a$ in state $s$
$\pi(s)$	deterministic policy: the action taken in state $s$
$\pi_\theta$	policy with parameters $\theta$
$V^\pi(s)$	state value: $\E_\pi[G_t \mid S_t = s]$
$Q^\pi(s, a)$	action value: $\E_\pi[G_t \mid S_t = s, A_t = a]$
$V^(s)$ , $Q^(s, a)$	optimal value functions
$A^\pi(s, a)$	advantage: $Q^\pi(s, a) - V^\pi(s)$
$\pi^*$	an optimal policy
$v_\pi, q_\pi$	Sutton & Barto's lowercase equivalents of $V^\pi, Q^\pi$ — we use the capital forms

Estimates carry hats or subscripted parameters: $\hat{V}$ , $V_\theta$ , $Q_w$ . Target-network parameters are $\theta^-$ (Chapter 10). Time-indexed estimates during learning are $V_t$ , $Q_t$ .

4. Learning Quantities

Symbol	Meaning
$\alpha$	step size / learning rate
$\epsilon$	exploration probability (ε-greedy)
$\delta_t$	TD error: $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$
$e_t(s)$ or $z_t$	eligibility trace (Chapter 7)
$\lambda$	trace-decay / λ-return parameter
$\rho_t$	importance-sampling ratio $\frac{\pi(A_t \mid S_t)}{b(A_t \mid S_t)}$
$b(a \mid s)$	behavior policy (off-policy learning)
$\mathcal{D}$	replay buffer or offline dataset
$J(\theta)$	policy performance objective, $\E_{\pi_\theta}[G_0]$
$\nabla_\theta J$	policy gradient
$\hat{A}_t$	advantage estimate (e.g. GAE, Chapter 11)
$\eta$ , $\beta$ , $\tau$	temperature / trust-region / soft-update coefficients, per chapter

5. Probability and Analysis

Symbol	Meaning
$\E[X]$ , $\E_\pi[\cdot]$	expectation (under policy $\pi$ )
$\mathrm{Var}[X]$	variance
$X \sim p$	$X$ is distributed according to $p$
$\mathcal{N}(\mu, \sigma^2)$	Gaussian distribution
$\KL(p \,\\|\, q)$	Kullback–Leibler divergence
$\mathcal{H}(p)$	entropy
$\\|f\\|_\infty$	sup-norm, $\max_s \lvert f(s) \rvert$
$\mathcal{T}^\pi$ , $\mathcal{T}^*$	Bellman expectation / optimality operators (Chapter 4)
$\mathbb{1}[\cdot]$	indicator function
$x^\top y$	inner product; vectors are columns
$\mathcal{O}(\cdot)$ , $\tilde{\mathcal{O}}(\cdot)$	asymptotic complexity (ignoring log factors)

6. Conventions

Expectation subscripts. $\E_\pi[\cdot]$ abbreviates the expectation over trajectories generated by $\pi$ interacting with the MDP: $S_0 \sim \mu_0$ , $A_t \sim \pi(\cdot \mid S_t)$ , $S_{t+1} \sim P(\cdot \mid S_t, A_t)$ . When only one random variable matters we name it explicitly, e.g. $\E_{a \sim \pi(\cdot \mid s)}$ .
Episodic vs. continuing. Episodic tasks terminate at $T$ ; we treat termination as entering an absorbing state with zero reward, so a single set of equations covers both cases (Chapter 3).
Greedy operators. $\argmax_a Q(s, a)$ breaks ties arbitrarily unless stated otherwise.
Gradients. $\nabla_\theta$ denotes the gradient with respect to $\theta$ ; when the variable is clear from context we write $\nabla$ .
Code. Code samples use NumPy for tabular methods and PyTorch for function approximation. Variables in code mirror the math: gamma, alpha, td_error, q_values.
Citations. Papers are cited inline by first author and year, with a link, and collected with annotations in the bibliography.

7. Deep-RL and Robotics Notation (Parts II–IV)

Symbol	Meaning
$\theta$ , $\phi$ , $\psi$	parameters of policy, value/critic, and model networks respectively
$\theta^-$ , $\phi'$	target-network parameters
$L(\theta)$	loss function
$\pi_\theta(a \mid s)$	parameterized stochastic policy
$\mu_\theta(s)$	parameterized deterministic policy (DDPG/TD3)
$Q_\phi(s, a)$ , $V_\phi(s)$	parameterized critics
$\hat{p}_\psi$	learned dynamics model
$z_t$ , $h_t$	latent state / recurrent state of a world model (Chapter 23)
$o_{1:t}$	observation history
$g$	goal (goal-conditioned RL, Chapter 19)
$\pi(a \mid s, g)$	goal-conditioned policy
$r_\phi(x, y)$	reward model over prompt–response pairs (Chapter 20)
$a_{t:t+H}$	action chunk over horizon $H$ (Chapter 24)

When a chapter needs a symbol not listed here, it defines the symbol at first use and stays consistent within the chapter. If you ever meet an undefined symbol, that is a bug — the notation contract of this book is that everything traces back to this page.