Deivation Inequalities II: From the Chernoff Bounds to the Bounded-Difference Inequality

36-465/665, Spring 2021

18 February 2021 (Lecture 6)

\[ \newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]} \newcommand{\Risk}{r} \newcommand{\EmpRisk}{\hat{r}} \newcommand{\Loss}{\ell} \newcommand{\OptimalStrategy}{\sigma} \DeclareMathOperator*{\argmin}{argmin} \newcommand{\ModelClass}{S} \newcommand{\OptimalModel}{s^*} \DeclareMathOperator{\tr}{tr} \newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}} \newcommand{\myexp}[1]{\exp{\left( #1 \right)}} \]

Last time

We want to give high-probability bounds on how far \(Z\) is from \(\Expect{Z}\)
- Ultimately we want to show \(\Prob{|\EmpRisk(\hat{s}) - \Risk(\hat{s})| \geq g(n,\alpha)} \leq \alpha\), or \(\Prob{|\Risk(\hat{s}) - \Risk(s^*)| \geq h(n,\alpha)} \leq \alpha\) but we’ve got to build up to that
Markov’s inequality: \(\Prob{Z \geq \epsilon} \leq \Expect{Z}/\epsilon\)
Markov \(\Rightarrow\) Chebyshev: \(\Prob{|Z-\Expect{Z}| \geq \epsilon} \leq \Var{Z}/\epsilon^2\)
\(\Rightarrow\) If \(X_i\) are IID then \(\Prob{|\overline{X}_n - \Expect{X}|\geq \epsilon} \leq \Var{X}/n\epsilon^2\)
Exponential Markov/Chernoff bounds: \(\Prob{Z \geq \epsilon} \leq \min_{t > 0}{e^{-t\epsilon} \Expect{e^{tZ}}}\) and \(\Prob{Z \leq \epsilon} \leq \min_{t > 0}{e^{t\epsilon} \Expect{e^{-tZ}}}\)
\(Z\) is sub-Gaussian \(\Leftrightarrow\) \(\Expect{e^{tZ}} \leq e^{t\Expect{Z}} e^{t^2 c^2/2}\) for some \(c>0\)
If \(X_i\) are IID and sub-Gaussian, then \(\Prob{|\overline{X}_n - \Expect{X}|\geq \epsilon} \leq 2\myexp{-n\epsilon^2/2c^2}\)

This time

Deviation inequalities for sums/averages of independent but not identically distributed (INID) variables
Deviation inequalities for functions of multiple random variables which are not sums or averages

Sums of INID variables

If \(X_1\) and \(X_2\) are independent, then \(\Expect{f(X_1) g(X_2)} = \Expect{f(X_1)}\Expect{g(X_2)}\) for any functions \(f, g\)
\(\therefore\) \(\Expect{e^{t(X_1+X_2)}} =\Expect{e^{tX_1} e^{tX_2}} = \Expect{e^{t X_1}} \Expect{e^{t X_2}}\) if \(X_1\) and \(X_2\) are independent
\(\therefore\) \(M_{X_1+X_2}(t) = M_{X_1}(t) M_{X_2}(t)\) if \(X_1\) and \(X_2\) are independent
If \(X_1\) and \(X_2\) are independent sub-Gaussians with constants \(c_1^2\), \(c_2^2\), then \(X_1+X_2\) is sub-Gaussian with constant \(c_1^2+c_2^2\) \[ M_{X_1+X_2}(t) = M_{X_1}(t) M_{X_2}(t) \leq e^{t\Expect{X_1}} e^{t\Expect{X_2}} e^{t^2 c_1^2/2} e^{t^2 c_2^2/2} \]
If \(X_i\) are all sub-Gaussian with constants \(c_i^2\) and expectations \(\mu_i\), and they’re all independent, then \[ \Prob{\left|\sum_{i=1}^{n}{X_i} -\sum_{i=1}^{n}{\mu_i}\right|\geq \epsilon} \leq 2\myexp{-\frac{\epsilon^2}{2\sum_{i=1}^{n}{c_i^2}}} \]
Off-line:
- What’s the deviation bound for the average (not sum) of the \(X_i\)?
- How do the deviation bounds for the sum and average simplify if all the \(c_i^2 \leq C^2\) for some \(C>0\)?

Hoeffding’s inequality

Suppose \(Z \in [a, b]\) with probability 1, with expectation \(\mu\)

Hoeffding’s bound: \(M_{Z}(t) \leq e^{t\mu} e^{t^2(b-a)^2/8}\)

Proof: basically that \(\Var{Z} \leq (b-a)^2/4\), plus some extra probability to argue that this upper-bounds the MGF
- Details not very important, in online handout to be posted on class website
Implication: a bounded random variable is sub-Gaussian, with \(c^2 = (b-a)^2/4\)

Hoefdding’s inequality (2)

Suppose \(Z_i \in [a_i, b_i]\), with expectation \(\mu_i\)
For each \(Z_i\), \(M_{Z_i}(t) \leq e^{t\mu_i} e^{t^2(b_i - a_i)^2/8}\)
- Proof: previous slide
Hoeffding’s bound or Hoeffding’s inequality: \[ \Prob{\left|\sum_{i=1}^{n}{Z_i - \mu_i}\right| \geq \epsilon} \leq 2 \myexp{-\frac{\epsilon^2}{2\sum_{i=1}^{n}{(b_i - a_i)^2/4}}} \]
Proof: Multiply \(M_{Z_i}\) together to get the MGF of the sum, then apply the sub-Gaussian bound
- i.e., combine the previous two slides

Hoeffding’s inequality (3)

Suppose \(Z_i \in [a_i, b_i]\), with expectation \(\mu_i\)
Define \(\overline{Z}_n \equiv n^{-1}\sum_{i=1}^{n}{Z_i}\)
Define \(\overline{\mu}_n \equiv n^{-1}\sum_{i=1}^{n}{\mu_i}\)
Hoeffding’s inequality or Hoeffding’s bound: \[ \Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon} \leq 2\myexp{-\frac{n^2\epsilon^2}{2\sum_{i=1}^{n}{(b_i - a_i)^2/4}}} \]
- Proof: if the average is \(\epsilon\) away from its expectation, the sum must be \(n\epsilon\) away from its expectation; apply previous slide
Numerator in the exponent is \(\propto n^2\), denominator has \(n\) terms, typically we get an \(O(n)\) rate

Example: log loss

Say we’re doing classification, so we’re predicting \(Y=0\) or \(Y=1\), based on \(X\), we think \(\Prob{Y=1|X=x} = q(x)\) but really \(\Prob{Y=1|X=x} = p(x)\)
\(Z_i =\) the log loss we experience on data point \((X_i, Y_i)\) \[\begin{eqnarray} Z_i & = & -Y_i \log{q(X_i)} - (1-Y_i) \log{(1-q(X_i))}\\ & = & -\log{(1-q(X_i))} - Y_i \log{\frac{q(X_i)}{1-q(X_i)}}\\ \Expect{Z_i|X=x} & = & -\log{(1-q(x))} - p(x) \log{\frac{q(x)}{1-q(x)}} \end{eqnarray}\]
- Note how the log odds ratio shows up
  - This is why logistic regression is such a big deal for classification…

Example: log loss (2)

Given \(X_i=x_i\), each \(Z_i\) can take only 1 of 2 values
\(\therefore\) for each \(Z_i\) we have \(|b_i - a_i| = \left|\log{\frac{q(x_i)}{1-q(x_i)}}\right|\)
Assume \(|\log{\frac{q(x_i)}{1-q(x_i)}}| \leq w\)
- “logit-bounded” or “bounded log-odds ratio”
\(\therefore\) \[ \Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon \mid| X_1,\ldots X_n} \leq 2\myexp{-\frac{n^2 \epsilon^2}{\frac{1}{2}n w^2}} \]
Notice \(X_i\) don’t appear in the bound, so \[ \Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon} \leq 2\myexp{-2\frac{n \epsilon^2}{w^2}} \]
Conclusion: if we use the log loss for classification, and the log odds ratio is bounded, the empirical risk is exponentially unlikely to be far from the true risk
- … for a particular fixed model
  - We’ll see in later classes how to make this bound apply to every model in some big class at once
- This result is also useful in many other situations with binary data, not just classification (de-noising binary signals or images, social network analysis, …)

What about functions that aren’t sums or averages?

One idea: Taylor series / linearization:
- if each \(X_i \approx \mu_i + \Delta_i\)
- and the \(\Delta_i\) are all small,
- and \(\partial f/\partial x_i\) is small
  - or, really, if the product \(\Delta_i \partial f/\partial x_i\) is small
- then \(f(X_1, \ldots X_n) \approx f(\mu_1, \ldots \mu_n) + \sum_{i=1}^{n}{\Delta_i (\partial f/\partial x_i)}\)
- This is a sum and we know how to give deviation bounds for it
Drawback: we’re back to asymptotic approximations again
Idea we can take away from this: if every input to the function can only change it a little, and the inputs are independent, then there should be a deviation inequality

The Efron-Stein Inequality

Suppose \(X_1, \ldots X_n\) are independent but not necessarily identically distributed, and \(Z=f(X_1, \ldots X_n)\). Say \(Y_1, \ldots Y_n\) are independent copies of the \(X_i\), and define \(Z_i=f(X_1, \ldots X_{i-1}, Y_i, X_{i+1}, \ldots X_n)\). Then \[ \Var{Z} \leq \frac{1}{2}\sum_{i=1}^{n}{\Expect{(Z-Z_i)^2}} \]

Proof: a bit complicated and uses a special probability technique (“martingales”), so see notes that’ll be on the class website
- A key step is more intuitive: \(\Var{X} = \frac{1}{2}\Expect{(X-Y)^2}\) where \(Y\) is an independent copy of \(X\)
Moral: randomly changing one input at a time doesn’t change the function too much \(\Rightarrow\) randomly changing all the inputs independent can’t change the function too much

The bounded difference property

A function \(f(x_1, \ldots x_n)\) has the bounded difference property when changing just one input can only make a limited difference to the function: \[ |f(x_1, \ldots, x_{i-1}, x_i, x_{i+1}, \ldots x_n) - f(x_1, \ldots x_{i-1}, y_i, x_{i+1}, \ldots x_n)| \leq d_i \]
Some simple examples:
- \(\sin{\left(\sum_{i=1}^{n}{w_i x_i}\right)}\), with \(d_i=2\) for all \(i\)
- \(\mathrm{logit}^{-1}\left(\sum_{i=1}^{n}{\beta_i x_i}\right)\) with \(d_i=1\) for all \(i\)
- We will see more examples later

Bounded difference + Efron-Stein

Remember we’re writing \(Z=f(X_1, \ldots X_n)\), and \(Z_i = f(X_1, \ldots Y_i, \ldots X_n)\) where \(Y_i\) has the same distribution as \(X_i\) but is independent
Bounded difference property \(\Rightarrow\) \(\Expect{(Z-Z_i)^2} \leq d_i^2\)
Thus \(\Var{Z} \leq \frac{1}{2}\sum_{i=1}^{n}{d_i^2}\)
Now use Chebyshev: \[ \Prob{|Z-\Expect{Z}|\geq\epsilon} \leq \frac{1}{2}\frac{\sum_{i=1}^{n}{d_i^2}}{\epsilon^2} \]
Can we do better if we assume more?

Hoeffding + bounded differences = McDiarmid

The bounded difference or McDiarmid inequality

If \(f\) has the bounded difference property, then \[ \Prob{|Z-\Expect{Z}|\geq\epsilon} \leq 2\myexp{-\frac{2\epsilon^2}{\sum_{i=1}^{n}{d_i^2}}} \]

Proof: Again, see handout online if you’re interested
Moral: If changing each input can only alter the function by so much, and the inputs are independent, then probability of large deviations is exponentially small

Making the bounded-differences inequality useful

Here’s the bounded-differences inequality again \[ \Prob{|Z-\Expect{Z}|\geq\epsilon} \leq 2\myexp{-\frac{2\epsilon^2}{\sum_{i=1}^{n}{d_i^2}}} \]
- Remember \(Z=f(X_1, \ldots X_n)\) and changing \(X_i\) (alone) alters \(Z\) by at most \(d_i\)
If \(\sum_{i=1}^{n}{d_i^2} = O(n)\) this isn’t helpful because the bound gets weaker as \(n\) grows
- Example: \(f(X_1, \ldots X_n) = \sum_{i=1}^{n}{X_i}\) with the \(X_i\)’s bounded (as in the sum form of Hoeffding’s bound)
- Example: \(\sin{\left(\sum_{i=1}^{n}{\beta_i X_i}\right)}\) with the \(X_i\)’s whatever they want to be (but \(\beta_i\) fixed as \(n\) changes)
If the \(d_i = O(1/n)\) then \(\sum_{i}{d_i^2} = nO(1/n^2) = O(1/n)\) and this is useful
- Example: averaging bounded random variables (as in the mean form of Hoeffding)

Example: Empirical vs. true distribution

Say the \(X_i\) are IID on the real line, with true CDF \(F^*\)
Define the empirical CDF as \[ \hat{F}_n(a) = \frac{1}{n}\sum_{i=1}^{n}{\Indicator{X_i \leq a}} \]
Note: \(\Expect{\hat{F}_n(a)} = F^*(a)\) (for all \(a\), \(n\))
Bounded difference property: for each \(i\), \(a\), changing \(X_i\) can change \(\hat{F}_n(a)\) by at most \(1/n\), so \(d_i=1/n\)
Implication: For any \(a \in \mathbb{R}\), \[ \Prob{\left|\hat{F}_n(a) - F^*(a)\right| \geq \epsilon} \leq 2\myexp{-2n\epsilon^2} \]

Example: Empirical vs. true distribution (2)

Define the maximum deviation as \[ M_n \equiv \max_{a}{\left|\hat{F}_n(a) - F^*(a)\right|} \]
Bounded difference property: Changing any one \(X_i\) can change \(M_n\) by at most \(1/n\), so \(d_i=1/n\) again
- Because that’s the most it can change any of the \(\hat{F}_n(a)\)
So \[ \Prob{\left|M_n - \Expect{M_n}\right| \geq \epsilon} \leq 2 \myexp{-2n\epsilon^2} \]
If we can show that \(\Expect{M_n} \rightarrow 0\), then we’ll have shown the whole function \(\hat{F}_n \rightarrow F^*\)
- This is called “uniform convergence”

Summing up

Moment-generating functions multiply when we add independent variables \(\Rightarrow\) we can handle sums and averages that aren’t IID
- Especially if everything stays sub-Gaussian
Hoeffding: bounded random variables are sub-Gaussian
Bounded difference property: changing one input to a function can’t change the output by more than a certiain amount
BDP \(\Rightarrow\) bounded variance (Efron-Stein)
BDP \(\Rightarrow\) sub-Gaussian-style deviation inequality (bounded difference or McDiarmid inequality)

Going forward:

Can we get more results that will hold for many different functions?
- Think of the bounded-difference inequality, but weaker or just different properties on the functions
- Concentration inequalities; next week
Can we get results that hold for infinitely many functions at once?
- Like the bound on \(M_n\) two slides ago
- Uniform convergence results; week after next
Can we please get back to learning prediction strategies?
- Immediately after that, I promise!