$$\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]} \newcommand{\Risk}{r} \newcommand{\EmpRisk}{\hat{\Risk}} \newcommand{\Loss}{\ell} \newcommand{\OptimalStrategy}{\sigma} \DeclareMathOperator*{\argmin}{argmin} \newcommand{\ModelClass}{S} \newcommand{\OptimalModel}{s^*} \DeclareMathOperator{\tr}{tr} \newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}} \newcommand{\myexp}[1]{\exp{\left( #1 \right)}} \newcommand{\eqdist}{\stackrel{d}{=}} \newcommand{\Rademacher}{\mathcal{R}} \newcommand{\EmpRademacher}{\hat{\Rademacher}} \newcommand{\Growth}{\Pi} \newcommand{\VCD}{\mathrm{VCdim}} \newcommand{\OptDomain}{\Theta} \newcommand{\OptDim}{p} \newcommand{\optimand}{\theta} \newcommand{\altoptimand}{\optimand^{\prime}} \newcommand{\ObjFunc}{M} \newcommand{\outputoptimand}{\optimand_{\mathrm{out}}} \newcommand{\Hessian}{\mathbf{h}} \newcommand{\Penalty}{\Omega} \newcommand{\Lagrangian}{\mathcal{L}} \newcommand{\HoldoutRisk}{\tilde{\Risk}} \DeclareMathOperator{\sgn}{sgn} \newcommand{\Margin}{M} \newcommand{\CumLoss}{L} \newcommand{\EnsembleAction}{\overline{a}} \newcommand{\CumEnsembleLoss}{\overline{\CumLoss}} \newcommand{\Regret}{R}$$

Previously

• States $y$, actions $a$, loss $\Loss(y,a)$, information $x$, strategies $s: x \mapsto a$, model class $\ModelClass$
• Assume $(X,Y)$ are IID so $\Risk(s) = \Expect{\Loss(Y, s(X))}$ is well-defined and is what we’d really want to know
• Risk is forward-looking (expected loss on new data) and probabilistic (expected loss on new data)
• IID data assumption can be weakened to other kinds of stochastic process (say, Markov chains)
• But “risk” just doesn’t make sense without some stochastic / probability assumptions

Low regret learning

• Regret of an action $a$ in state $y$ is $$\Loss(y,a) - \min_{a^{\prime}}{\Loss(y,a^{\prime})}$$ $=$ extra loss we could have avoided by doing something else
• Regret of a strategy $s$ on state sequence $y_{1:n}$ and information sequence $x_{1:n}$, compared to model class $\ModelClass$, is $$\sum_{t=1}^{n}{\Loss(y_t, s(x_{1:t}))} - \min_{s^{\prime} \in \ModelClass}{\sum_{t=1}^{n}{\Loss(y_t, s^{\prime}(x_{1:t}))}}$$ $=$ extra loss we could have avoided by following a different strategy

Risk vs. Regret

• Risk:
  • Forward-looking
  • Assumes a probability distribution
  • Absolute
• Regret
  • Backward-looking
  • Only involves the actual path, not a distribution over paths (no probability)
  • Comparative (relative to $\ModelClass$)
• Regret makes sense when dealing with
  • Lots of uncertainty about the right model
  • Lots of uncertainty about the distribution and whether it changes
  • Intelligent adversaries (“adversarial” learning, not the same as “adversarial examples”)
• How can we achieve low regret?

The game (“prediction with expert advice”)

• We start round $t$ with weights $w_{i,t-1}$ over the experts $i$
• We observe $x_t$
• Every expert gets to announce $s_i(x_{1:t})$, for short $s_i(t)$
• We take action $\EnsembleAction_t$ based on the experts’ advice and the weights
• We see the actual $y_t$ and calculate all the losses
• We update the weights, to get $w_{i,t}$

Regret of the ensemble (vs. the best expert)

• The retrospectively-best expert, at time $n$, is $$\OptimalModel = \argmin_{i \in 1:q}{\CumLoss_{i,n}}$$
• The regret of the ensemble, compared to this best expert, is $$\Regret_n \equiv \CumEnsembleLoss_n - \min_{i \in 1:q}{\CumLoss_{i,n}}$$
• We would like this to be small, but what’s “small” here?

Sub-linear regret

• Assume $\Loss \in [0,1]$
  • If $\Loss \in [0,m]$, work with $\frac{\Loss}{m}$ and pick up an over-all factor of $m$ at the end
• The worst the regret could possible be is $\Regret_n = n$
• An idiot who picks one action and sticks with it no matter what has a regret that’s $O(n)$ (at most)
• $\therefore$ It’d be good if a non-idiot could beat that
• $\therefore$ We want sub-linear regret, $\Regret(n) = o(n)$
• An ensemble which always has $\Regret(n) = o(n)$ is called Hannan consistent
• Hannan consistency implies $n^{-1}\Regret(n) = o(1)$
  • Confusingly, this is sometimes called “vanishing regret”

A regret bound

If $\Loss(y,a) \in [0,1]$ and is convex in $a$, and we use multiplicative weight training with learning rate $\beta$ over $q$ experts, then $$\Regret_n \leq \frac{n\beta}{8} + \frac{\log{q}}{\beta}$$ and, with the right choice of $\beta$, $$\Regret_n \leq \sqrt{\frac{n}{2}\log{q}}$$

Proving the regret bound

• Introduce $W_t = \sum_{i=1}^{q}{w_{i,t}}$, with $W_0 = q$
• We’re going to show that $\log{W_n/W_0}$ is lower bounded by the loss of the best expert, and upper bounded by the loss of the ensemble
  • The difference will tell us about the regret

Lower bound

Tricks, modifications

• $\Loss \in [0,m]$: work with $\frac{\Loss}{m}$, multiply the final bound by $m$
• Time-varying loss functions: no change necessary
• If $\Loss(y,a)$ isn’t convex in $a$: Randomly pick expert $i$ with probability $u_{i,t-1}$ and take action $s_i(t)$; then $\Expect{\Loss(y, A_t)}$ is convex in the probabilities, and the bounds hold for expected regret
  • Expectation is over the random choices of expert, not over the data
  • Can you use Hoeffding’s inequality to argue the regret is small with high probability?
• Unbounded losses: can work if the loss function has nice properties, e.g., log probability loss
• Infinitely many experts: needs some extra structure, like an initial distribution over experts
  • Multiplicative weight training then becomes evolution by natural selection (without mutation or other sources of variation)

Unbounded horizons, changing learning rates

• With fixed $\beta$ the regret bound is linear
• If we know $n$, we can pick $\beta$ to get a regret bound of $\sqrt{\frac{n}{2}\log{q}}$ which is sub-linear
• But the right $\beta$ depends on $n$, what if we want to keep going?
• A similar (but even longer) proof shows that if $\beta_t = \sqrt{8\frac{\log{q}}{t}}$, then $$\Regret_n \leq 2\sqrt{\frac{n}{2}\log{q}} + \sqrt{\frac{\log{q}}{8}}$$
• We have to slow down learning ($\beta_t \rightarrow 0$) but we still get sub-linear regret, and basically just pick up a factor of 2