\[ \newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} \newcommand{\indep}{\perp} \]

“State”

• In classical physics or dynamical systems, “state” is the present variable which fixes all future observables

• Back away from determinism: state determines distribution of future observables

• Would like a small state

• Would like the state to be well-behaved, e.g. Markov

• Try construct states by constructing predictions

Notation etc.

• Upper-case letters are random variables, lower-case their realizations

• Stochastic process \(\ldots, X_{-1}, X_0, X_1, X_2, \ldots\)

• \(X_{s}^{t} = (X_s, X_{s+1}, \ldots X_{t-1}, X_t)\)

• Past up to and including \(t\) is \(X^t_{-\infty}\), future is \(X_{t+1}^{\infty}\)
  • Discrete time isn’t required but cuts down on measure theory

Making a Prediction

• Look at \(X^t_{-\infty}\), make a guess about \(X_{t+1}^{\infty}\)
• Most general guess is a probability distribution
• Only ever attend to selected aspects of \(X^t_{-\infty}\)
  • mean, variance, phase of 1st three Fourier modes, ….
• \(\therefore\) guess is a function or statistic of \(X^t_{-\infty}\)
• A good statistic should give us lots of information about \(X_{t+1}^{\infty}\)

Entropy

• The entropy of a random variable \(X\): \[ H[X] \equiv -\sum_{x}{\Prob{X=x}\log_2{\Prob{X=x}}} \]
• Some mathematical properties:
  • \(H[X] \geq 0\)
  • \(H[X] = 0\) if and only if \(\Prob{X=x_0} = 1\) for some \(x_0\)
  • If \(X\) has \(k\) possible values, \(H[X] \leq \log_2{k}\)
  • \(H[X] = \log_2{k}\) if and only if \(\Prob{X=x} = 1/k\) for all \(x\)
• Data processing inequality: \(H[X] \geq H[\sigma(X)]\)
  • \(H[X] = H[\sigma(X)]\) iff \(\sigma\) is 1-1
• \(H[X] \approx\) number of bits needed to encode the value of \(X\)
• \(H[X]\) is the uncertainty in the distribution of \(X\)

Joint and Conditional Entropy

• Conditional entropy: \[\begin{eqnarray} H[Y|X=x] & \equiv & -\sum_{y}{\Prob{Y=y|X=x}\log_2{\Prob{Y=y|X=x}}}\\ H[Y|X] & \equiv & \Expect{H[Y|X=X]} \end{eqnarray}\]
  • Condition on multiple variables in the natural way
• Joint entropy: \[ H[X,Y] = H[X] + H[Y|X] = H[Y] + H[X|Y] \]
• Same interpretations as entropy

Entropy Rate

• Limiting entropy per time-step: \[ h \equiv \lim_{n\rightarrow\infty}{\frac{1}{n}H[X_1^n]} \]
• Equivalently, limiting conditional entropy: \[ h = \lim_{n\rightarrow\infty}{H[X_{n+1}|X_1^n]} \]
• Both limits exist and are equal for all stationary processes
• Interpretations:
  • How many bits do we need to code the next observation, given the complete history? (“source coding”)
  • How much uncertainty is there about what happens next, given the complete history?
  • \(h=0\) for deterministic processes

Information

• The information \(X\) gives about \(Y\) is the reduction in uncertainty: \[ H[Y] - H[Y|X] \]
• Some algebra shows that \[ H[Y] - H[Y|X] = H[X] - H[X|Y] \]
• Define the mutual information \[ I[X;Y] \equiv H[Y] - H[Y|X] \]
• Some properties:
  • \(I[X;Y] \geq 0\)
  • \(I[X;Y] = 0\) if and only if \(X\) and \(Y\) are independent
• Data processing: \(I[X;Y] \geq I[\sigma(X);Y]\)
  • Can be preserved even if \(\sigma\) isn’t 1-1

Predictive Information

• We want to make predictions
• Use \(I[X^{\infty}_{t+1};\sigma(X_{-\infty}^t)]\) to measure how much information the statistic \(\sigma\) gives us about the future
  • Can also look at \(I[X^{t+m}_{t+1};\sigma(X_{-\infty}^t)]\) for all \(m\)
• How big can we make that?

Predictive Sufficiency

• For any statistic \(\sigma\), \[ I[X^{\infty}_{t+1};X_{-\infty}^t] \geq I[X^{\infty}_{t+1};\sigma(X_{-\infty}^t)] \]
• \(\sigma\) is predictively sufficient iff \[ I[X^{\infty}_{t+1};X_{-\infty}^t] = I[X^{\infty}_{t+1};\sigma(X_{-\infty}^t)] \]
• Sufficient statistics retain all predictive information in the data

Predictive states

(Crutchfield and Young 1989)

• Histories \(a\) and \(b\) are equivalent iff \[ \Prob{X_{t+1}^{\infty}|X_{-\infty}^{t} = a} = \Prob{X_{t+1}^{\infty}|X_{-\infty}^{t} = b} \]
• \([a] \equiv\) all histories equivalent to \(a\)
• The statistic of interest, the predictive state, is \[ \epsilon(x^t_{-\infty}) = [x^t_{-\infty}] \]
• Set \(s_t = \epsilon(x^t_{-\infty})\)
• A state is an equivalence class of histories and a distribution over future events
• IID = 1 state, periodic = \(p\) states

set of histories, color-coded by conditional distribution of futures

Partitioning histories into predictive states

Sufficiency

(Shalizi and Crutchfield 2001)

• \[ I[X^{\infty}_{t+1};X^t_{-\infty}] = I[X^{\infty}_{t+1};\epsilon(X^t_{-\infty})] \]
• because \[\begin{eqnarray*} \Prob{X^{\infty}_{t+1}|S_t = \epsilon(x^t_{-\infty})} & = & \int_{y \in [x^t_{-\infty}]}{\Prob{X^{\infty}_{t+1}|X^t_{-\infty}=y} \Prob{X^t_{-\infty}=y|S_t = \epsilon(x^t_{-\infty})} dy}\\ & = & \Prob{X^{\infty}_{t+1}|X^t_{-\infty}=x^t_{-\infty}} \end{eqnarray*}\]

A non-sufficient partition of histories

Effect of insufficiency on predictive distributions

Markov Properties

• Future observations are independent of the past given the causal state: \[ X^{\infty}_{t+1} \indep X^{t}_{-\infty} | S_{t+1} \]
• by sufficiency: \[\begin{eqnarray*} \Prob{X^{\infty}_{t+1}|X^{t}_{-\infty}=x^t_{-\infty}, S_{t+1} = \epsilon(x^t_{-\infty})} & = & \Prob{X^{\infty}_{t+1}|X^{t}_{-\infty}=x^t_{-\infty}}\\ & = & \Prob{X^{\infty}_{t+1}|S_{t+1} = \epsilon(x^t_{-\infty})} \end{eqnarray*}\]

Recursive Updating/Deterministic Transitions

• Recursive transitions for states: \[ \epsilon(x^{t+1}_{-\infty}) = T(\epsilon(x^t_{-\infty}), x_{t+1}) \]
  • Automata theory: “deterministic transitions” (even though there are probabilities)
• In continuous time: \[ \epsilon(x^{t+h}_{-\infty}) = T(\epsilon(x^t_{-\infty}),x^{t+h}_{t}) \]

Predictive States are Markovian

• \[ S_{t+1}^{\infty} \indep S^{t-1}_{-\infty}|S_t \]
• because \[ S_{t+1}^{\infty} = T(S_t,X_{t}^{\infty}) \] and \[ X_{t}^{\infty} \indep \left\{ X^{t-1}_{-\infty}, S^{t-1}_{-\infty}\right\} | S_t \]

Minimality

• \(\epsilon\) is minimal sufficient \(=\) can be computed from any other sufficient statistic
• \(\therefore\) for any sufficient \(\eta\), exists a function \(g\) such that \[ \epsilon(X^{t}_{-\infty}) = g(\eta(X^t_{-\infty})) \]
• \(\therefore\) if \(\eta\) is sufficient \[ I[\epsilon(X^{t}_{-\infty}); X^{t}_{-\infty}] \leq I[\eta(X^{t}_{-\infty}); X^{t}_{-\infty}] \]

Sufficient, but not minimal, partition of histories

Coarser than the predictive states, but not sufficient

Uniqueness

• There is really no other minimal sufficient statistic
• If \(\eta\) is minimal, there is an \(h\) such that \[ \eta = h(\epsilon) ~\mathrm{w.p.1} \]
• but \(\epsilon = g(\eta)\) (w.p.1)
• so \[\begin{eqnarray*} g(h(\epsilon)) & = & \epsilon\\ h(g(\eta)) & = & \eta \end{eqnarray*}\]
• \(\therefore\) \(\epsilon\) and \(\eta\) partition histories in the same way (w.p.1)

Minimal stochasticity

• If \(R_t = \eta(X^{t-1}_{-\infty})\) is also sufficient, then \[ H[R_{t+1}|R_t] \geq H[S_{t+1}|S_t] \]
• \(\therefore\) the predictive states are the closest we get to a deterministic model, without losing power

Entropy Rate

• \[\begin{eqnarray*} h & \equiv & \lim_{n\rightarrow\infty}{H[X_{n+1}|X^n_1]}\\ & = & \lim_{n\rightarrow\infty}{H[X_{n+1}|S_n]} ~\text{(sufficiency)}\\ & = & H[X_2|S_1] ~\text{(stationarity)} \end{eqnarray*}\]
• so the predictive states lets us calculate the entropy rate
  • and do source coding

Minimal Markovian Representation

• The observed process \((X_t)\) is non-Markovian and ugly
• But it is generated from a homogeneous Markov process \((S_t)\)
• After minimization, this representation is (essentially) unique
  • Can exist smaller Markovian representations, but then always have distributions over those states
  • and those distributions correspond to predictive states

What Sort of Markov Model?

• Common-or-garden HMM: \[ S_{t+1} \indep X_{t+1}|S_t \]
• But here \[ S_{t+1} = T(S_t, X_{t+1}) \]
• This is a chain with complete connections (Onicescu and Mihoc 1935; Iosifescu and Grigorescu 1990)

HMM

CCC

Example of a CCC: Even Process

• Blocks of As of any length, separated by even-length blocks of Bs
  • Histories ending \(AB\) have different implications than histories ending \(ABB\) than \(ABBB\) than \(ABBBB\)
  • \(\Rightarrow\) Not Markov at any finite order

How Broad Are These Results?

Knight (1975) gave most general constructions

• Non-stationary \(X\)
• \(t\) continuous (but discrete works as special case)
• \(X_t\) with values in continuous spaces
  • Lusin space = image of a complete separable metrizable space under a measurable bijection
• \(S_t\) is a Markov process with recursive updating

Connecting to Data

• Everything so far has been math/probability
  • The Oracle tells us the infinite-dimensional distribution of \(X\)
• Can we do some statistics and find the states?
• Two senses of “find”: learn in a fixed model vs. discover the right model

Learning

Problem: Given states and transitions (\(\epsilon, T\)), realization \(x_1^n\), estimate \(\Prob{X_{t+1}=x|S_t=s}\)

• Just estimation for stochastic processes
  • Easier than ordinary HMMs because \(S_t\) is a function of trajectory
  • Exponential families in the all-discrete case, very tractable

Discovery

Problem: Given \(x_1^n\), estimate \(\epsilon, T, \Prob{X_{t+1}=x|S_t=s}\)

• Much harder!
  • Why should this be possible at all?
  • Can’t cover every possible approach (and will neglect Langford, Salakhutdinov, and Zhang (2009);Pfau, Bartlett, and Wood (2010) both)

CSSR: Causal State Splitting Reconstruction

• Key observation: Recursion + one-step-ahead predictive sufficiency \(\Rightarrow\) general predictive sufficiency
  • Get next-step distribution right by independence testing
  • Then make states recursive

• Assumes discrete observations, discrete time, finite causal states

• Paper: Shalizi and Klinkner (2004); code, [https://github.com/stites/CSSR]

One-Step Ahead Prediction

• Start with all histories in the same state
• Given current partition of histories into states, test whether going one step further back into the past changes the next-step conditional distribution
  • Use a hypothesis test to control false positive rate
• If yes, split that cell of the partition, but see if it matches an existing distribution
  • Must allow this merging or else lose minimality
• If no match, add new cell to the partition
• Stop when no more divisions can be made or a maximum history length \(\Lambda\) is reached
  • For consistency, \(\Lambda < \frac{\log{n}}{h_1 + \iota}\) for some \(\iota\)

Ensuring Recursive Transitions

• Need to determinize a probabilistic automaton
• Several ways of doing this, but a solved technical problem in automata theory
  • Textbooks: Lewis and Papadimitriou (1998); Hopcroft and Ullman (1979)
  • Brilliant but bizarre math approach: Conway (1971)
• Trickiest coding in the algorithm and can influence the finite-sample behavior

Convergence

• \(\mathcal{S} =\) true predictive state structure
• \(\widehat{\mathcal{S}}_n\) = structure reconstructed from \(n\) data points
• Assume: finite # of states, every state has a finite history, using long enough histories, technicalities: \[ \Prob{\widehat{\mathcal{S}}_n \neq \mathcal{S}} \rightarrow 0 \]
• \(\mathcal{D} =\) true distribution, \(\widehat{\mathcal{D}}_n\) = inferred \[ \Expect{{\|\widehat{\mathcal{D}}_n - \mathcal{D}\|}_{1}} = O(n^{-1/2}) \]
  • Same order of convergence as for IID samples

Hand-waving

• Empirical conditional distributions for histories converge
  • ergodic theorem for Markov chains
• Histories in the same state become harder to accidentally separate
• Histories in different states become harder to confuse
• Each state’s predictive distribution converges \(O(n^{-1/2})\)
  • CLT for Markov chains

Example: The Even Process

reconstruction with \(\Lambda = 3\), \(n=1000\), \(\alpha = 0.005\)

N.B., CSSR did not know that there were 2 states, or how they were connected — it discovered this

Some Uses

• Neural spike train analysis (Haslinger, Klinkner, and Shalizi 2010), fMRI analysis (Merriam, Genovese and Shalizi in prep.)
• Geomagnetic fluctuations (Clarke, Freeman, and Watkins 2003)
• Natural language processing (M. Padró and Padró 2005b, 2005a, 2005c, 2007a, 2007b)
• Anomaly detection (David S. Friedlander, Phoha, and Brooks 2003; Davis S. Friedlander et al. 2003; Ray 2004)
• Information sharing in networks (Klinkner, Shalizi, and Camperi 2006; Shalizi, Camperi, and Klinkner 2007)
• Social media propagation (Cointet, Faure, and Roth 2007)

Summary

• Your stochastic process has a unique, minimal Markovian representation
• This representation has nice predictive properties
• Can reconstruct from sample data in some cases
  • and a lot more could be done in this line

Backup: Why Care About Sufficiency?

• Optimal strategy, under any loss function, only needs a sufficient statistic (Blackwell & Girshick)
• Strategies using insufficient statistics can generally be improved (Blackwell & Rao)
• Excuse for not worrying about particular loss functions

Backup: A Cousin: The Information Bottleneck

(Tishby, Pereira, and Bialek 1999)

• For inputs \(X\) and outputs \(Y\), fix \(\beta > 0\), find \(\eta(X)\), the bottleneck variable, maximizing \[ I[\eta(X);Y] - \beta I[\eta(X);X] \]
• give up 1 bit of predictive information for \(\beta\) bits of memory
• Predictive sufficiency comes as \(\beta \rightarrow \infty\), unwilling to lose any predictive power

Backup: Extension to Input-Output Systems

(Littman, Sutton, and Singh 2002; Shalizi 2001)

• System output \((X_t)\), input \((Y_t)\)
• Histories \(x^t_{-\infty}, y^t_{-\infty}\) have distributions of output \(x_{t+1}\) for each further input \(y_{t+1}\)
• Equivalence class these distributions and enforce recursive updating
• Internal states of the system, not trying to predict future inputs

Backup: Statistical Forecasting Complexity

• Statistical forecasting complexity is (Grassberger 1986; Crutchfield and Young 1989) \[ C \equiv I[\epsilon(X^t_{-\infty});X^t_{-\infty}] \]
• \(=\) amount of information about the past needed for optimal prediction
• \(=H[\epsilon(X^t_{-\infty})]\) for predictive causal states
• \(=\) expected algorithmic sophistication (Gács, Tromp, and Vitanyi 2001)
• \(=\log\)(period) for period processes
• \(=\log\)(geometric mean(recurrence time)) for stationary processes
• Property of the process, not our models