\[ \newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]} \newcommand{\Risk}{r} \newcommand{\EmpRisk}{\hat{r}} \newcommand{\Loss}{\ell} \newcommand{\OptimalStrategy}{\sigma} \DeclareMathOperator*{\argmin}{argmin} \newcommand{\ModelClass}{S} \newcommand{\OptimalModel}{s^*} \DeclareMathOperator{\tr}{tr} \newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}} \newcommand{\myexp}[1]{\exp{\left( #1 \right)}} \]

Previously

• Risk of strategy \(s\) is \(\Risk(s) \equiv \Expect{\Loss(Y, s(X))}\)
• Optimal strategy \(\OptimalStrategy \equiv \argmin_{\text{all functions}\ s}{\Risk(s)}\)
• Best-in-class \(\OptimalModel \equiv \argmin_{s \in \ModelClass}{\Risk(s)}\)
• \(\Risk(s) = \Risk(\OptimalStrategy) + (\Risk(\OptimalModel) - \Risk(\OptimalStrategy)) + (\Risk(s) - \Risk(\OptimalModel))\)
• Empirical risk \(\EmpRisk(s) \equiv n^{-1}\sum_{i=1}^{n}{\Loss(Y_i, s(X_i))}\)
• Empirical risk minimizer \(\hat{s} = \argmin_{s \in \ModelClass}{\EmpRisk(s)}\)
  • Best performance, on the training data, within the class \(\ModelClass\)
• How well does \(\hat{s}\) actually do in the future?

Why we’re not quite there

• \(\hat{s}\) isn’t independent of the data: \[\begin{eqnarray} \Expect{\EmpRisk(\hat{s})} & = & \Expect{n^{-1}\sum_{i=1}^{n}{\Loss(Y_i, \hat{s}(X_i))}}\\ & = & \frac{1}{n}\sum_{i=1}^{n}{\Expect{\Loss(Y_i, \hat{s}(X_i))}} \end{eqnarray}\]
• Now the terms in the sum aren’t IID so this isn’t just \(\Risk(\hat{s})\)
  • Still identically distributed but not independent

What’s the issue, mathematically?

• Universal convergence: You can pick any one \(s \in \ModelClass\), and any \(\epsilon > 0\), and I can always find \(\alpha_{s}(n, \epsilon)\) which guarantees that \[ \Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon) \] with \[ \lim_{n\rightarrow \infty}{\alpha_{s}(n, \epsilon)} = 0 \]
• Meaning: every \(\EmpRisk(s)\) converges to its \(\Risk(s)\)
• but at its own rate (given by the \(\alpha_s\) function)
• Convergence might be really, really slow for some strategies \(s\)
• which’d be bad news if the optimal strategy \(\OptimalModel\) was really slowly converging

What’s the issue, mathematically? (2)

• Universal convergence: for every \(s \in \ModelClass\), you can pick any \(\epsilon > 0\), and I can find \(\alpha_{s}(n, \epsilon)\) which guarantees that \[ \Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon) \] with \[ \lim_{n\rightarrow \infty}{\alpha_{s}(n, \epsilon)} = 0 \]
• Versus uniform convergence: you can pick any \(\epsilon > 0\), and I can find a single \(\alpha(n, \epsilon)\) so that \[ \Prob{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon} \leq \alpha(n, \epsilon) \] with \[ \lim_{n\rightarrow\infty}{\alpha(n, \epsilon)} = 0 \]
• All the strategies converge at least that fast
  • Including the optimal strategy \(\OptimalModel\)

How would uniform convergence help?

• Let’s abbreviate: \[ \Gamma_n \equiv \max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \]
• Uniform convergence means \(\Gamma_n \rightarrow 0\) as \(n\rightarrow\infty\)
• Empirical risk vs. true risk: \[ |\EmpRisk(\hat{s}) - \Risk(\hat{s})| \leq \max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} = \Gamma_n \]
• \(\Rightarrow\) Knowing how fast \(\Gamma_n\) goes to 0 tells us how optimistic the empirical risk is

How would uniform convergence help? (2)

• True risk vs. optimal true risk: \[\begin{eqnarray} \Risk(\hat{s}) - \Risk(\OptimalModel) & = & \Risk(\hat{s}) - \EmpRisk(\OptimalModel) + \EmpRisk(\OptimalModel) - \Risk(\OptimalModel)\\ & \leq & \Risk(\hat{s}) - \EmpRisk(\hat{s}) + \EmpRisk(\OptimalModel) - \Risk(\OptimalModel)\\ & \leq & |\Risk(\hat{s}) - \EmpRisk(\hat{s})| + |\EmpRisk(\OptimalModel) - \Risk(\OptimalModel)|\\ & \leq & 2\max_{s}{|\EmpRisk(s) - \Risk(s)|} = 2\Gamma_n \end{eqnarray}\]
  • Works because \(\EmpRisk(\hat{s}) \equiv \min_{s \in \ModelClass}{\EmpRisk(s)} \leq \EmpRisk(\OptimalModel)\)
• \(\therefore\) Knowing how fast \(\Gamma_n \rightarrow 0\) tells us how close the empirical risk minimizer \(\hat{s}\) will be to the best-in-class strategy \(\OptimalModel\)

How could we get uniform convergence?

• Suppose the class of strategies \(\ModelClass\) was finite: \[ S = \left\{ s_1, s_2, \ldots s_q \right\} \] with \(q < \infty\)
• Then universal convergence \(\Rightarrow\) uniform convergence: \[\begin{eqnarray} \left\{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon \right\} & = & \left\{ \exists i \in 1:q ~\text{such that} ~ |\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon \right\}\\ \left\{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon \right\} & = & \cup_{i=1}^{q}{\left\{ |\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon \right\}}\\ \Prob{\Gamma_n \geq \epsilon} & \leq & \sum_{i=1}^{n}{\Prob{|\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon}}\\ & \leq & q \max_{i \in 1:q}{\Prob{|\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon}} \end{eqnarray}\]
• This is called the union bound
  • For any two events \(A\) and \(B\), \(\Prob{A \cup B} = \Prob{A} + \Prob{B} - \Prob{A \cap B} \leq \Prob{A} + \Prob{B}\)
• Extra simple when all the deviation probabilities on the RHS are the same
  • E.g. if the loss function \(\Loss\) is bounded

Why the union bound isn’t enough

• Most interesting strategy classes are infinite!
• \(\therefore\) union bound tells us the deviation probability for \(\Gamma_n < \infty\) which is not helpful
• Two ways to rescue this:
  1. Approximating an infinite \(\ModelClass\) by a finite one, then union bound + approximation inaccuracy
  2. Trickery: directly calculate (or bound) \(\Expect{\Gamma_n}\) and then argue \(\Gamma_n\) concentrates on its expected value

The route of approximation

• Fix \(\epsilon > 0\)
• Suppose we can find \(q(\epsilon)\) strategies \(s_1, \ldots s_q\) which approximate all the \(s\)’s to within \(\pm \epsilon\):
  • You get to name any \(s\) you like, and I find an \(s_i\) in the set where, no matter what:
  1. \(|\EmpRisk(s_i) - \EmpRisk(s)| \leq \epsilon\), and
  2. \(|\Risk(s_i) - \Risk(s)| \leq \epsilon\)
• The \(s_i\) cover \(\ModelClass\), and \(q(\epsilon)\) is the covering number of \(\ModelClass\) (at tolerance \(\epsilon\))
  • \(q(\epsilon)\) will grow as \(\epsilon \rightarrow 0\)
  • Think: how many little balls of radius \(\epsilon\) would you need to cover some geometric region?
  • Think of a finite-length interval, say \([0,1]\); how many points \(x_i\) in the interval do you need to pick to guarantee that \(|x_i-x| \leq \epsilon\)? How does that grow with \(\epsilon\)? What if it was a 2D area instead of a 1D interval? etc.
• Consequence: \[\begin{eqnarray} |\EmpRisk(s) - \Risk(s)| & = & |\EmpRisk(s) - \EmpRisk(s_i) + \EmpRisk(s_i) - \Risk(s_i) + \Risk(s_i) - \Risk(s) |\\ & \leq & |\EmpRisk(s) - \EmpRisk(s_i)| + |\EmpRisk(s_i) - \Risk(s_i)| + |\Risk(s) - \Risk(s_i)| \\ & \leq & \epsilon + |\EmpRisk(s_i) - \Risk(s_i)| + \epsilon\\ & = & 2\epsilon + |\EmpRisk(s_i) - \Risk(s_i)| \end{eqnarray}\]

The route of approximation (2)

• Just saw \[ |\EmpRisk(s) - \Risk(s)| \leq 2\epsilon + |\EmpRisk(s_i) - \Risk(s_i)| \] no matter what
• Now use the union bound: for large \(n\), \(\max_{i \in 1:q(\epsilon)}{|\EmpRisk(s_i) - \Risk(s_i)|} \leq \epsilon\) with high probability
• So if \(q(\epsilon) < \infty\), we can show \[ \max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \leq 3\epsilon \] with high probability
• \(\epsilon\) can be as small as we like so \(\Gamma_n \rightarrow 0\)

Pros and cons of the approximation / covering route

Pros

• If we can find a covering set, it will often work for all distributions of the data
• We get to do neat math about distances between functions
• Fundamental: if there is uniform convergence, then there (pretty much) must be covering sets with \(q(\epsilon) < \infty\)

Cons

• The union bound is rather loose
  • Deviations aren’t mutually exclusive across different \(s\)’s, there’s lots of overlap in those events!
• We need to use sophisticated math about function spaces

The route of trickery

• Suppose the loss function \(\Loss\) is bounded, \(0 \leq \Loss \leq m\)
• Then \(\EmpRisk(s)\) has the bounded-difference property; changing one data point can’t change \(\EmpRisk(s)\) by more than \(m/n\)
• and \(\Gamma_n = \max_{s}{|\EmpRisk(s) - \Risk(s)|}\) also has the bounded difference property with bound \(m/n\)
• and so \[ \Prob{\Gamma_n \geq \Expect{\Gamma_n} + \epsilon} \leq \myexp{-2n\frac{\epsilon^2}{m^2}} \]
• \(\therefore\) Try to calculate, or bound \(\Expect{\Gamma_n}\)
• Ideally without knowing any of the true risks…
• Subject for next week

Summing up

• We’ve worked on universal convergence of empirical risk to true risks: for each \(s \in \ModelClass\), \[ \Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon) \]
• To show that \(\EmpRisk(\hat{s}) \rightarrow \Risk(\hat{s})\) and that \(\Risk(\hat{s} \rightarrow \Risk(\OptimalModel)\), we want to control \(\Gamma_n \equiv \max_{s \in \ModelClass}{|\EmpRisk(s) - \Risk(s)|}\)
• Uniform convergence means: \[ \Prob{\Gamma_n \geq \epsilon} \leq \alpha(n, \epsilon) \]
• If \(\ModelClass\) were finite we could use the union bound to get uniform convergence from universal convergence
• If \(\ModelClass\) can be well-approximated by a finite number of models, we can use the union bound (“covering number” approach)
  • Theoretically basic but needs too much math
• For bounded losses, \(\Gamma_n\) concentrates on \(\Expect{\Gamma_n}\), so we’ll work on calculating / controlling \(\Expect{\Gamma_n}\)
  • This will turn out to be much more practical