$$\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}$$

Housekeeping

• No class on Thursday (enjoy Carnival)

Kernel machines (reprise)

• Functions of the form $$s(x) = \sum_{i=1}^{n}{\alpha_i K(x, x_i)}$$
  • In principle needn’t be a sum over the $n$ training points, but in practice that’s pretty much always what the ERM looks like

Kernels and their associated functions

• Kernel functions are inner products in a feature space: $$K(x, z) = \varphi(x) \cdot \varphi(z) = \sum_{j=1}^{\infty}{\varphi_j(x) \varphi_j(z)}$$
• Often convenient to use orthonormal basis functions, so $\phi_i \cdot \phi_j = \delta_{ij}$, in which case $$K(x,z) = \sum_{j=1}^{\infty}{\lambda_j \phi_j(x) \phi_j(z)}$$ and $\sqrt{\lambda_j}\phi_j = \varphi_j$
• The basis functions and the $\lambda_j$s are eigenfunctions and eigenvalues: $$\int{\phi_j(z) K(x, z) dz} = \lambda_j \phi_j(x)$$
  • We can choose the eigenfunctions so that they’re orthonormal

Kernel machines vs. basis expansions

• Use the eigenfunctions to re-write the kernel machine: $$\begin{eqnarray} s(x) & = & \sum_{i=1}^{n}{\alpha_i K(x, x_i)}\\ & = & \sum_{i=1}^{n}{\alpha_i \sum_{j=1}^{\infty}{\lambda_j \phi_j(x) \phi_j(x_i)}}\\ & = & \sum_{j=1}^{\infty}{\lambda_j \left(\sum_{i=1}^{n}{\alpha_i \phi_j(x_i)}\right) \phi_j(x)}\\ & = & \sum_{j=1}^{\infty}{\beta_j \phi_j(x)} \end{eqnarray}$$

• This is called an expansion of $s$ in the basis given by the $\phi_j$, with expansion coefficients (or just coefficients) $\beta_j$
• Kernel machines are implicitly using expansions in the basis of the eigenfunctions of the kernel

• This lets us use many more than $n$ basis functions while only having $n$ weights $\alpha_i$

Towards being more concrete about all this

• Helps to examine a particular data set
• The COMPAS data for Broward County, Florida ($\approx$ Ft. Lauderdale)
• 3 key attributes: age of arrestee; number of prior convictions; whether re-arrested (not convicted) for a violent offense within 2 years
  • Other attributes in the data set include sex, race, type of charge, …
• I’ve split the data into an 80% training set and a 20% testing set

Look at the data

Priors vs. age for the training set, color-coded for recidivism. Points are “jittered” so that multiple individuals with equal features aren’t superposed. The “rugs” along the axes give a sense of the marginal distributions of the two attributes for the two classes.

Bringing the kernel in

• For us, $X=$ (age, priors), and $Y=$ 0/1 indicator for re-arrest for violence
  • As usual, arrange into $[n\times p]$ matrix $\mathbf{x}$ and $[n \times 1]$ matrix $\mathbf{y}$
• We’ll look next time at using classifiers, but for now let’s treat this as a regression problem, so our prediction of $Y$ could be treated as a probability
• There are lots of analyses we could do…
• Almost all kernel machines need the kernel matrix, usually written $\mathbf{K}$ where $K_{ij} = K(x_i, x_j)$
  • Inner product, in feature space, of every data point with every other data point
  • Does not involve $\mathbf{y}$ at all (just like the “hat” matrix in linear regression)
• Kernel least squares (no regularization): $$\mathbf{\alpha} = \mathbf{K}^{-1}\mathbf{y}$$
• Kernel ridge regression (regularized): $$\mathbf{\alpha} = (\mathbf{K} + \lambda \mathbf{I})^{-1} \mathbf{y}$$

Picking a kernel

• We still need a kernel!
• A good default choice is the Gaussian kernel: $$K(x,z) = \myexp{-\|x-z\|^2/2h^2}$$
  • Still need to pick a bandwidth $h$; there are tricks

• One reason this is a good default choice is that it’s really using polynomials to all (even) orders $$\begin{eqnarray} \myexp{u} & = & \sum_{j=0}^{\infty}{\frac{u^j}{j!}}\\ \myexp{-u^2/2h^2} & = & \sum_{j=0}^{\infty}{\left(\frac{-1}{2h^2}\right)^j \frac{u^{2j}}{j!}} \end{eqnarray}$$

• This is also called the radial basis function kernel

The kernel matrix and (approximate) basis functions

• The kernel matrix is square, symmetric and non-negative-definite, so it has an eigendecomposition $$\mathbf{K} = \mathbf{v}^T \mathbf{d} \mathbf{v}$$
• $\mathbf{v} =$ matrix of eigenvectors, which are orthonormal so $\mathbf{v}^T = \mathbf{v}^{-1}$
• $\mathbf{d} =$ diagonal matrix of eigenvalues
• Column $j$ of $\mathbf{v} \approx$ the orthonormal basis function $\phi_j$ evaluated on the data points
• $d_{jj} \approx \lambda_j$, the corresponding eigenvalue

Visualizing the kernel matrix

• Displaying the full $[3216 \times 3216]$ matrix $\mathbf{K}$ is too much; pick a subset of 100 data points just for visualization

Visualizing the basis functions

• It’s easier to visualize the basis functions in 1D so we’ll just use age for demo purposes

Visualizing the basis functions

• It’s easier to visualize the basis functions in 1D so we’ll just use age for demo purposes

The first four eigenfunctions

What about the eigenvalues?

Scale the eigenvectors by the eigenvalues

Notice that lots of the eigenvectors don’t even show up any more, because they’re being multiplied by very small (square roots of) eigenvalues

Back to fitting

• I’ll do kernel ridge regression, with a rather arbitrarily-picked value of $\lambda$

Fits on the training set, with color indicating fitted value (darker being lower predictions of recidivism, lighter higher predictions), and shape indicating actual outcome

Look at the testing set